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ABSTRACT 


The  Keesler  USAFSAAS  was  tasked  to  evaluate  an  automated  course  planning  tool  developed  by 
Rand  known  as  MODIA  (Method  of  Designing  Instructional  Alternatives).  ATC/XPT  chaired  a 
working  group  composed  of  HQ  ATC,  Rand, and  Keesler  representatives  to  develop  evaluation 
objectives  for  this  project.  From  10  November  1975  to  4 June  1976,  the  MODIA  programs 
were  installed  and  operated  on  Keesler's  Honeywell  6060  computer.and  data  were  collected 
to  address  the  evaluation  objectives.  This  report  presents  results  and  conclusions  regard- 
ing MODIA 's  cost,  validity,  effectiveness,  usefulness,  limitations,  and  identifies  needed 
changes.  The  data  were  sufficient  to  conclude  that  MODIA  has  the  potential  to  be  an 
effective  planning  tool  whose  use  could  lead  to  more  cost-effective  technical  training 
courses.  However,  this  study  did  not  determine  the  procedures  by  which  MODIA  would  be 
operated  in  the  USAFSAAS,  thus  leaving  unanswered  questions  as  to  the  control  and  value 
of  the  system  to  SAAS  management. 

RECOMMENDATIONS 

a.  Recommend  the  Rand  Corporation,  with  assistance  from  Keesler,  modify  the  MODIA 
programs  to  resolve  the  problems  discussed  in  this  report. 

b.  Recommend  the  Keesler  USAFSAAS  be  tasked  to  conduct  a Phase  II  follow-on  evaluation 
of  the  MODIA  System  using  the  24  electronics  courses  associated  with  Hasty  Spark.  This 
evaluation  will  establish  both  the  procedures  and  organizational  configuration  necessary  to 
support  MODIA  planning,  and  using  realistic  management  problems,  establish  the  utility  of 
the  system  to  SAAS  management. 
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EVALUATION  OF  THE  MODIA  SYSTEM 


SUMMARY  REPORT 

OBJECTIVES: 

MODIA,  or  the  Method  Of  Designing  Instructional  Alternatives,  is  a computerized 
course  planning  tooT  designed  by  the  Rand  Corporation  for  Air  Training  Command.  It  was 
evaluated  in  an  ATC  technical  training  environment  to  determine  the  validity  of  the  course 
operation  and  course  cost  data  it  generates,  to  assess  Its  effectiveness  as  a planning 
tool,  to  determine  its  implementation  and  potential  operating  costs,  and  to  determine  to 
which  types  of  formal  technical  training  the  system  can  be  applied.  An  additional  objective 
of  this  evaluation  was  to  determine  what  changes.  If  any,  should  be  made  to  MODIA  to 
improve  its  overall  utility  in  this  environment. 


APPROACH: 

Six  courses,  five  from  Keesler  and  one  from  Lowry,  were  selected  for  Involvement  in 
the  service  test.  To  assess  MODIA's  validity,  ISD  teams  representing  each  of  the  Keesler 
courses  in  conjunction  with  a MODIA  Interface  Team  used  MODIA  to  design  their  respective 
courses.  The  simulated  course  operation  provided  by  MODIA  was  then  compared  for  congruity 
with  known  information  about  the  existing  courses.  MODIA's  effectiveness  as  a planning 
tool  was  evaluated  by  having  the  ISD  teams  design  two  alternatives  to  each  existing  course. 
These  alternatives  were  then  compared  with  the  existing  course  on  factors  such  as  course 
cost,  resource  utilization,  student  queuing,  facility  and  equipment  requirements. 

Costs  associated  with  full  MODIA  implementation  and  operation  were  estimated 
from  cost  data  collected  durinq  the  service  test,  From  this  figure  was  subtracted  the 
estimated  cost  of  the  conventional  planning  process  to  give  a rough  estimate  of  the 
increased  cost  of  MODIA.  To  determine  MODIA's  applicability  and  usefulness  for  planning 
Types  1-6  courses,  the  scope  and  purpose  of  each  type  of  planning  was  compared  to  MODIA's 
capability.  Further,  a survey  of  the  course  planning  done  at  this  center  from  1970-1975 
was  conducted  to  estimate  the  magnitude  of  the  planning  effort  which  could  be  supported 
by  MODIA.  Finally,  interviews  were  conducted  with  the  participating  ISD  teams  to  assess 
any  perceived  limitations  with  the  system  and  to  determine  the  overall  reactions  of 
planners  to  MODIA. 


RESULTS  AND  CONCLUSIONS: 

Course  operation  and  course  cost  parameters  for  four  of  the  five  courses  simulated 
with  MODIA  were  very  close  to  the  parameters  of  the  actual  ongoing  courses.  The  one 
discrepancy  occurred  when  planners  attempted  to  simulate  Keesler 's  Course  3AQR30020-1 , 
Electronic  Principles.  Because  this  course  Is  so  large,  the  MODIA  system  could  not  accom- 
modate all  the  planning  factors  and  full  simulation  of  course  operation  was  not  achieved. 
However,  based  upon  MODIA's  performance  with  the  other  four  courses,  evaluators  concluded 
that  MODIA's  outputs  are  accurate  and  valid. 

For  three  of  the  five  Keesler  courses  which  were  revised  with  MODIA,  projected  five- 
year  cost  savings  were  realized.  These  savings  ranged  from  $48,100  for  3ABR30434  to 
$173,300  for  3ABR30331 . Results  of  the  study  support  many  of  the  Rand  claims  for  MODIA. 
Planners  were  able  to  rapidly  manipulate  many  design  factors  to  create  alternative  courses 
which  could  then  be  compared  on  dimensions  such  as  overall  cost,  resource  utilization, 
trainer  requirements  and  so  forth.  Planners  were  very  effective  In  eliminating  any  wait 
for  resources  and  yet  reducing  overall  course  costs.  Because  MODIA  provides  the  Impact  of 
a design  decision  on  all  other  decisions,  evaluators  concluded  that  MODIA  stimulates  more 
thorough  and  Innovative  approaches  to  course  design  and  thus  makes  possible  the  creation 
of  more  cost-effective  courses. 


Concerning  system  usefulness,  evaluators  concluded  that  MODIA  could  not  be  used  for 
Types  1,  5,  and  6 planning,  would  have  limited  use  for  Types  2 and  4 planning,  and  could 
be  used  extensively  for  Type  3 planning.  Based  upon  data  collected  during  the  service 
test,  Keesler  revises  and  implements  on  the  average  approximately  42,500  hours  of  Type  3 
instruction  annually.  This  figure  equates  to  approximately  80%  of  all  the  course  hours 
revised  and  implemented  at  this  center  each  year.  Thus,  MODIA  could  be  a very  useful 
technical  training  planning  tool. 

MODIA  first-year  cost,  determined  from  the  cost  data  collected  during  the  service 
test  and  projected  to  the  population  of  42,500  hours  of  Type  3 instruction,  was  estimated 
at  approximately  $800,000.  This  figure  includes  computer  charges  for  planning  two  complete 
course  revisions  plus  personnel  costs.  The  cost  of  conventional  planning  for  the  analogous 
portion  of  the  ISD  process  was  estimated  at  $312,000  for  the  same  number  of  course  hours. 
This  latter  figure  suffers  from  unreliability,  and  the  report  discusses  reasons  for  the 
large  difference.  Nevertheless,  evaluators  concluded  that  the  increased  cost  of  MODIA 
would  be  offset  by  the  savings  accrued  from  more  cost-effective  instructional  systems. 

Interface  Team  observations  and  interviews  with  ISD  team  members  identified  several 
changes  which  should  be  made  to  MODIA  to  improve  its  acceptance  and  efficiency.  These 
changes  range  from  minor  modifications,  such  as  not  having  the  option  to  cancel  certain 
MODIA  printouts,  to  three  relatively  major  changes.  First,  MODIA  occasionally  garbled 
certain  inputs  which  forced  planners  to  repeat  certain  phases  of  the  design  process,  and 
this  problem  needs  to  be  eliminated.  Second,  MODIA  program  limitations  prohibit  planners 
from  designating  more  than  31  resource  groups,  and  several  ISD  team  members  believed  this 
number  should  be  increased.  Finally,  MODIA  users  requested  a program  change  to  permit  the 
calculation  of  resource  utilization  when  a course  revision  is  implemented  for  a program 
which  is  already  in  progress. 

ADOPTION: 

Because  of  the  narrow  focus  of  the  service  test  plus  the  limited  number  of  courses  to 
which  MODIA  was  applied,  system  adoption  cannot  be  recommended  at  this  time.  Questions 
remain  as  to  how  School  policies  and  objectives  will  be  incorporated  into  the  planning 
effort,  who  will  make  the  choice  among  the  alternatives  designed,  and  how  resource 
distribution  across  a group  of  homogeneous  courses  can  be  accomplished.  Further,  although 
the  study  implied  that  MODIA  planning  might  lead  to  manpower  reductions,  the  procedures 
and  organizational  configuration  necessary  to  support  MODIA  planning  were  not  addressed. 
More  extensive  and  realistic  use  of  the  system  is  recommended  to  answer  these  questions. 
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MOD  I A EVALUATION 
SERVICE  TEST  PROGRAM 

SECTION  A - INTRODUCTION 


1.  Authority:  This  evaluation  program  was  conducted  under  the  authority  of  the  Plan  for 
Evaluating  the  MOOIA  System,  6 April  1976  and  ATC/XP  letter  to  Keesler/XR,  10  October 
1974,  Support  Requirements  for  Evaluation  of  MODIA  (Method  of  Designing  Instructional 
A1 ternatives) . 


Objectives: 


a.  Determine  the  cost  of  using  the  MODIA  System  for  planning  training  courses  in  a 
technical  training  environment. 


b.  Assess  the  validity  of  the  MODIA  System  to  determine  the  degree  to  which  the 
system  and  each  of  its  components  does  what  it  is  supposed  to  do. 


c.  Assess  MODIA’s  effectiveness  as  a planning  tool. 


d.  Determine  the  potential  usefulness  of  MODIA  in  terms  of  planning  Types  1,2, 3, 4, 5, 
and  6 courses. 


e.  Determine  what  changes  to  MODIA  might  improve  its  effectiveness. 

3.  Background:  The  Method  of  Designing  Instructional  Alternatives  (MODIA)  System  was 
developed  by  Rand  Corporation  for  Air  Training  Corrmand.  It  is  a tool  designed  to 
faciliate  course  planning.  Its  aim  is  to  provide  a systematic  process  for  planning  the 
mix  of  students,  instructors,  materials,  equipment  and  facilities, and  procedures  by 
which  all  of  these  elements  work  together  to  effect  student  mastery  of  the  subject  matter, 
and  to  provide  planners  with  consistent  estimates  of  the  cost  and  performance  of  alternative 
plans  for  a training  course.  A critical  design  review  of  the  MODIA  System  was  conducted 
during  October  1973.  The  purpose  of  that  review  was  to  have  technical  training  planners 
try  out  the  preliminary  MODIA  System  model  so  that  Rand  designers  could  gain  the  insights 
necessary  to  develop  the  final  operational  system.  Keesler  and  Lowry  personnel  partic- 
ipated in  that  review  and  furnished  information  since  used  by  Rand  to  accomplish  system 
changes  which  are  considered  sufficient  to  satisfy  Air  Training  Command  needs.  Four  major 
steps  embraced  the  MODIA  project  at  Keesler:  (1)  installing  and  debugging  the  MODIA 
System  computer  programs  in  the  H-6060  computer  at  Keesler,  (2)  training  the  MODIA 
Interface  Team  members  and  evaluation  team  members,  (3)  shaking  down  the  system,  and  (4) 
evaluating  MODIA.  Rand  personnel  installed  and  debugged  three  MODIA  System  computer 
programs  in  the  Keesler  H-6060  computer  during  the  period  10  November  - 20  November  1975. 
These  programs  were  Installed  in  an  operational  configuration  which  required  them  to  be 
operated  on  a time-sharing  basis  with  WWMCCS  progranmer  and  operator  courses  during  the 
service  test  period.  Training  of  the  MODIA  Interface  Team  was  conducted  during  the 
period  1-12  December  1975.  Shakedown  of  the  MODIA  System  was  subsequently  accomplished 
by  exercising  MODIA  on  Courses  3ABR32831  and  3ABR30430.  The  requirement  to  train  the  ISD 
teams  for  these  two  courses,  in  addition  to  affording  shakedown  data,  enabled  Interface 
Team  members  to  gain  proficiency  in  using  MODIA.  The  shakedown  of  MODIA  was  completed 
on  5 March  1976. 
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SECTION  B - METHOD 


4.  Experimental  Design: 

a.  Course  Selected:  Six  courses  (five  from  Keesler  and  one  from  Lowry)  were 
selected  for  involvement  in  the  service  test.  Specific  courses  selected  were: 

3ABR32834 , Avionic  Inertial  and  Radar  Nav  Sys  Specialist 

3ABR30434,  Ground  Radio  Equipment  Repairman 

3ABR30331,  ATC  Raoar  Repairman 

3ABR30333,  Automatic  Tracking  Radar  Repairman 

3AQR30020-1,  Electronic  Principles 

3AIR75100-X,  Instructor  Training 

Course  selection  was  based  on  the  following: 

(1)  Each  needed  critical  review  of  cost  effectiveness. 

(2)  Low,  medium,  and  high  student  loads  were  represented. 

(3)  Basic  electronics  and  principles-centered  electronic  technology  training  were 
represented. 


(4)  Different  instructional  approaches  were  represented. 

(5)  Stability  of  materiel  varied  from  rapid  changes  to  long  life. 


The  assumption  was  that  these  courses  reflected  the  best  planning  possible  through  conven- 
tional means,  a prerequisite  deemed  necessary  to  compare  the  conventional  approach  with 
the  MODIA  System.  Keesler  courses  selected  had  progressed  through  all  steps  of  the  5-Step 
ISD  Model.  The  Lowry  Course  (3A1R751 00-X)  had  progressed  through  Step  3 of  the  5-Step 
ISD  Model . 


b.  ISD  Team:  Six  ISD  teams  were  identified  to  participate  in  the  evaluation  as 
course  planners.  Each  ISD  team  was  comprised  of  a curriculum  training  specialist,  branch 
training  specialist,  resource  training  specialist,  and  subject  matter  specialists.  Also 
School  Headquarters  staff  participants  (Plans  and  Resources  Branch  staff  personnel)  provided 
inputs  as  required.  Each  ISD  team  provided  the  planning  factors  to  exercise  the  MODIA 
System  during  the  service  test.  Specific  ISD  teams  were: 

ISD  Team  A - Course  3ABR32834 

ISD  Team  B - Course  3ABR30434 

ISD  Team  C - Course  3ABR30331 


ISD  Team  D 
ISD  Team  E 
ISD  Team  F 


Course  3ABR30333 
Course  3AQR30020-1 
Course  3AIR75100-X 
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c.  MODIA  Interface  Team:  The  Interface  Team  served  as  an  interface  between  MODIA 
and  the  course  planners.  They  operated  the  MODIA  System,  assisted  course  planners  in 
studying  alternatives,  and  aided  course  planners  in  Interpreting  MODIA  outputs  to 
determine  the  best  instructional  alternatives. 

d.  Data  Collection:  To  evaluate  MODIA,  performance  data  (results  of  exercising  MODIA 
to  develop  instructional  alternatives  for  the  courses  selected  for  the  study)  and  partic- 
ipant opinions  collected  by  structured  Interviews  were  analyzed.  Two  different  Interview 
questionnaires,  one  for  Interface  Team  members  and  one  for  the  ISD  teams,  were  utilized. 
Supplemental  cost  data  were  collected  from  the  five  Keesler  service  test  ISD  teams  which 
reflected  the  best  estimate  of  the  cost  of  planning  manually.  With  these  data,  planning 
costs  for  the  MODIA  and  conventional  approaches  were  compared.  Structured  interviews 
were  conducted  at  the  end  of  the  evaluation  period.  The  performance  data  were  collected 
by  observations  and  by  analyzing  MODIA  outputs  while  using  the  system  to  achieve  assigned 
planning  tasks.  Further,  data  reflecting  the  number  of  course  hours  that  had  been  revised 
and/or  newly  prepared  at  Keesler  during  the  period  1970-1975  were  collected  from  the 
files  of  the  Plans  Branch  (TTOX)  to  assess  MODIA's  potential  usefulness. 

e.  Questionnaires:  The  questionnaires  used  in  the  structured  interviews  are  discussed 
below: 

(1)  Interview  Questionnaire  for  the  Interface  Team:  This  questionnaire  solicited 
the  opinions  of  MODIA  Interface  Team  members  concerning  the  ease  or  difficulty  of  operating 
MODIA,  the  ease  or  difficulty  of  Interpreting  MODIA  outputs,  and  MODIA's  effectiveness 

as  a planning  tool.  A sample  copy  of  this  questionnaire  (with  raw  data  summaries)  is 
contained  in  Appendix  1. 

(2)  Interview  Questionnaire  for  the  ISD  Teams:  This  questionnaire  solicited 
the  opinions  of  each  ISD  team  concerning  the  ease  or  difficulty  of  providing  the  planning 
data  for  exercising  MODIA,  the  ease  or  difficulty  of  interpreting  MODIA  results,  the 
ease  or  difficulty  to  plan  different  approaches  to  instruction,  the  completeness  and 
comprehensiveness  of  the  information  provided  by  MODIA,  acceptance  of  MODIA  as  a planning 
tool,  and  the  type  courses  to  which  MODIA  planning  could  be  applied  in  a technical  train- 
ing environment.  A sample  copy  of  this  questionnaire  (with  raw  data  suumaries)  is 
contained  in  Aopendix  2. 

E 

Data  contained  in  the  questionnaires  were  consolidated  and  analyzed  to  compare 
participant  acceptance  of  MODIA  with  the  conventional  mode  of  planning;  also,  the  data 
were  used  to  assess  the  effectiveness  of  MODIA  as  a planning  tool. 

f.  History  Sheet:  A background  questionnaire  provided  the  evaluation  team  with  the 
skill  level  and  experience  of  the  service  test  participants.  A copy  of  this  questionnaire 
is  contained  in  Appendix  3. 

5.  Analysis  Procedures:  The  procedures  employed  to  analyze  the  data  collected  during  the 
study  were  as  follows: 

a.  MODIA  System  Cost:  The  cost  of  MODIA  as  a planning  system  was  determined  as 
i follows: 

(1)  Data  had  already  been  collected  from  each  of  the  service  test  ISD  teams  which 
reflected  the  manhours  expended  in  completing  that  part  of  Step  4 of  the  ISD  Model  (for 
which  MODIA  was  applied)  in  planning  the  current  courses. 

(2)  Data  reflecting  cost  in  facilities,  equipment,  manhours,  materials,  and 
funds  was  summarized  to  determine  the  USAF  cost  for  Implementing  and  operating  MODIA  at 
Keesler  during  the  service  test.  The  MODIA  implementation  cost  was  prorated  against  all 
applicable  Keesler  courses.  The  operating  cost  was  extrapolated  to  the  five  service 
test  courses. 
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(3)  Acceptance:  Structured  interviews  were  used  to  collect  data  for  determining 
MODIA  acceptance  by  course  planners  in  a technical  training  environment.  A Chi  Square 
test  was  used  to  analyze  the  data  to  determine  if  there  was  a difference  in  the  participants* 
opinions  about  the  use  of  MODIA  versus  the  conventional  mode  of  planning  training  courses. 

The  acceptance  of  any  real  difference  was  at  the  .05  level  of  significance. 

d.  MODIA  System  Changes  to  Improve  Effectiveness:  Participant  opinions  were  collected 
by  structured  interviews  to  solicit  recoumended  changes  for  improving  MODIA 
effectiveness  in  a technical  training  environment. 


SECTION  C - RESULTS 


6.  General:  This  section  summarizes  the  data  collected  during  the  service  test  (8  March  - 
4 June  1976). 

7.  MODIA  System  Problems/Constraints  Encountered:  During  the  shakedown  period,  the 
following  system  limitations  were  discovered  through  trial  and  error: 

a.  The  design  of  the  User  Interface  (UI)  program  permitted  only  20  objectives  to  be 
used  at  any  one  time  in  planning  an  instructional  program.  This  limitation  caused  the 
planners  to  design  the  courses  on  a block-by-block  basis  rather  than  the  entire  course 
on  one  run  of  the  MODIA  UI  program.  Also,  resource  specifications  limited  planners  to 
identifying  21  special  (equipment,  etc.)  and  10  other  (instructors,  rooms,  etc.)  resources. 
To  work  around  this  limitation,  course  planners  and  Interface  Team  members  had  to  group 
equipment  for  each  student  laboratory  position.  Rand  designers  were  advised  of  these 
limitations  during  January  1976  and  were  requested  to  furnish  the  MODIA  Interface  Team 

the  limitations  and  specifications  on  all  phases  of  the  User  Interface  (UI)  program  so 
that  MODIA  could  be  exercised  with  greater  efficiency  during  the  service  test  period.  The 
MODIA  Interface  Team  also  requested  Rand  personnel  expand  the  number  of  objectives 
that  could  be  applied  to  the  User  Interface  (UI)  program  to  250  and  increase  the  quantity 
of  resources  that  could  be  assigned  during  instructional  alternative  design.  The  limits 
on  entries  to  the  UI  were  furnished  by  Rand  in  February  1976  and  the  program  changes  were 
incorporated  in  April  1976  which  allowed  course  planners  to  use  250  objectives. 

b.  During  the  early  part  of  the  service  test.  Interface  and  ISD  team  members 
experienced  garbling  (malassignment)  of  resources  when  these  resources  were  allocated 
to  learning  events  during  the  MODIA  resource  contraints  phase  of  the  instructional 
design.  To  work  around  this,  course  planners  and  Interface  Team  members  had  to  try, 
through  trial  and  error,  various  combination  of  numbers  for  resources  allocated  to 
learning  events  until  a number  was  found  that  the  User  Interface  program  would  accept. 

When  the  correct  number  was  accepted,  the  computer  would  assign  the  resources  correctly 
to  the  appropriate  learning  events.  This  problem  persisted  occasionally  throughout  the 
service  test  program.  The  inability  of  course  planners  to  enter  discretionary  numbers 
for  resources  was  frustrating  at  times  and  forced  them  to  go  through  the  resource 
constraint  phase  two  or  three  times  to  secure  the  proper  assignment  of  resources  to 
learning  events. 

c.  Another  occasional  problem  which  persisted  throughout  the  service  test  was  that 
user  entries  at  the  keyboard  were  misread  by  the  User  Interface  (UI)  program.  Examples 
of  this  error  are  as  follows: 

INPUT  FROM  TERMINAL  COMPUTER  READ  (TERMINAL  PRINTOUT) 


168 

143 

243 

1680 

111/222 
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Error  in  Entry 
Error  in  Entry 
Error  in  Entry 
1114222 
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This  type  error  was  not  as  serious  as  the  garbling  of  resources  since  it  did  not  affect 
the  assignment  of  resources  to  learning  events  or  have  any  serious  effect  on  course 
design.  When  the  errors  occurred,  users  were  able  to  correct  them  later  in  the  course 
design  when  asked  at  the  end  of  the  UI  if  there  were  any  changes  to  be  made.  Therefore, 
this  malfunction  was  not  a serious  hindrance  during  the  service  test  period  of  8 March  - 
4 June  1976. 

8.  Background  of  Service  Test  Participants:  Table  1 provides  the  grade,  present  position, 
curriculum  and  teaching  experience  in  a technical  training  environment,  and  years  in 
present  position  for  the  service  test  participants.  The  average  time  spent  in  present 
position  and  the  average  curriculum  and  teaching  experience  for  all  participants  are  also 
shown. 

The  instructors  and  instructor  supervisors  served  as  subject  matter  experts.  The 
training  specialist  provided  guidance  on  school  policy  regarding  training  for  each  of  the 
Air  Force  specialties  selected  for  involvement  in  the  service  test.  The  MODIA  Interface 
Team  members  served  as  an  interface  between  MODIA  and  the  course  planners,  operated  the 
system  to  assist  course  planners  in  studying  alternatives,  and  aided  course  planners  in 
interpreting  MODIA  outputs  to  determine  the  best  course  design. 

9.  MODIA  System  Cost: 

a.  Comparison  of  MODIA  and  Manual  Planning:  The  MODIA  System  was  exercised  on  five 
Keesler  courses  to  determine  its  cost  as  a planning  tool  related  to  the  cost  of  the 
current  planning  system.  To  accomplish  this,  data  (facilities,  equipment,  manhours,  and 
material)  reflecting  the  USAF  cost  incurred  in  operating  MODIA  (see  Appendix  5)  during 
the  planning  of  each  Keesler  course  were  collected  and  compared  with  data  (manhours  and 
materials)  reflecting  the  cost  of  manually  planning  the  last  revision  to  each  course 
(see  Appendix  4).  No  data  exist  on  the  cost  of  doing  Step  4 under  the  ISD  process. 
Therefore,  the  manhours  attributed  to  the  manual  planning  process  were  derived  from 
professional  estimates  by  personnel  directly  responsible  for  the  last  revision  of  each 
Keesler  course  selected  for  the  service  test.  In  spite  of  the  subjectivity  of  the  data 
gathering  process,  the  general  uniformity  of  the  data  from  five  separate  ISD  teams  lends 
considerable  creditability  to  it.  Further,  the  number  of  hours  used  to  design  an 
instructional  alternative  using  MODIA  does  not  include  the  number  of  hours  to  input  the 
baseline  program  (present  instructional  system).  The  baseline  courses  were  entered  into 
the  computer  by  course  planners  during  MODIA  familiarization  sessions.  With  baseline 
data  established,  course  planners  were  able  to  compare  existing  instructional  alternatives 
with  MODIA-developed  alternatives.  However,  in  normal  operation,  course  planners  could 
get  cost  factors  for  existing  courses  from  the  comptroller  and  might  not  need  to 
establish  a baseline  in  MODIA  for  comparison  purposes.  Thus,  the  manhours  for  MODIA 
planning  reflect  the  creation  of  one  alternative  but  not  the  establishment  of  the  baseline. 
Also,  MODIA  Interface  Team  member  manhours  were  charged  against  the  MODIA  planning  process. 
This  was  done  because  the  plan  called  for  them  to  work  with  each  ISD  team  and. operate  MODIA 
to  plan  instructional  alternatives.  Based  upon  data  in  Appendices  4 and  5,  the  number  of 
manhours  required  to  plan  each  hour  of  instruction  and  the  cost  per  hour  of  planning 
effort  were  arrived  at  and  are  depicted  in  Appendices  7 and  8. 
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Table  2 provides  a comparison  of  the  average  cost  of  planning  each  hour  of  Instruction 
with  MODIA  and  the  average  cost  of  planning  each  hour  of  instruction  manually. 


TABLE  2 

COST  OF  PLANNING  ONE  INSTRUCTIONAL  ALTERNATIVE,  MODIA  VS  MANUAL 


Average  Cost/Hour 
Planning  Effort 

Range 

Manual  Course  Revision 

J7.331 

$6.62-$8. 11 

MODIA  Course  Revision 

$3.79* 

$3.46-$4.04 

$8.33^ 

$6.72-$10.22 

1 Manhour  cost  only  (excludes  administrative  support  cost). 

^ Manhour  + computer  support  cost  (excludes  administrative  support  and  implementation 
cost) . 


Results  indicate  that  the  total  MOOIA  planning  cost  is  slightly  higher  than  the 
current  manual  planning  cost.  However,  MODIA  affords  a significant  reduction  in  manhours 
required  (lower  manhour  cost)  when  planning  one  alternative  design  from  an  existing 
instructional  system. 

b.  MODIA  Implementation  Cost:  The  cost  of  implementing  MODIA  in  support  of  the  service 
test  is  shown  in  Appendix  6.  This  cost,  $36,080.25,  includes  USAF  cost  incurred  in 
facilities,  equipment,  materials,  computer  operation,  and  manhours  during:  (1)  installation 
and  debugging  of  the  system  on  the  H-6060  computer,  (2)  training  of  the  MODIA  Interface 
Team,  and  (3)  shakedown  of  the  system. 

10.  MODIA  System  Validation:  The  MODIA  System  was  exercised  to  measure  both  its  utility 
as  a planning  tool  in  a technical  training  environment  and  the  fidelity  of  its  products  as 
compared  to  actual  operating  courses.  Utility  was  gauged  by  the  ability  of  MODIA  to  accept 
and  process  all  the  factors  needed  to  plan  and  construct  a technical  training  course  with- 
in ATC  policies  and  regulations.  Members  of  all  five  ISO  teams  found  that  the  data  needed 
to  plan  courses  could  be  accommodated  by  MODIA  in  a systematic  fashion  and  no  gaps  existed 
in  the  programs. 

To  assess  MODIA's  validity,  ISD  teams  and  the  MODIA  Interface  Team  compared  salient 
aspects  of  five  operating  courses  at  Keesler  with  identical  aspects  of  these  courses  as 
simulated  by  MODIA.  For  each  course,  teams  exercised  MODIA  using  actual  student  entry 
data,  rate  of  student  failures,  quantity  of  resources  (instructors,  rooms,  hardware,  and 
test  equipment),  and  time  of  achievement  for  learning  objectives  based  upon  one  year's 
operation  of  the  actual  course.  As  can  be  seen  from  Tables  3 through  7,  MODIA  utilized 
this  data  to  produce  courses  with  parameters  very  similar  to  those  of  the  operating  courses. 
The  three  cost  figures  for  the  ongoing  courses  - Cost  Per  Student  Week,  Cost  Per  Graduate, 
and  First  Year  Cost  of  Course1  - were  obtained  from  the  Keesler  Comptroller.  Based  upon 
their  recommendation,  cost  data  provided  by  MODIA  is  considered  sufficiently  valid  if  it  is 
within  +10%  of  the  comptroller's  cost  figure.  Paragraphs  associated  with  each  table 
discuss  the  cost  differences  experienced. 


1.  See  Appendix  9 for  definition  of  cost  factors  used  to  determine  course  cost. 
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a.  Course  3ABR32834,  Avionic  Inertial  and  Radar  Navigation  System  Specialist:  This  is 
a group/lock-step  instructional  system  employing  510  hours  of  instruction.  Table  3 reflects 
results  obtained  by  ISO  Team  A.  System  fidelity,  as  evidenced  by  data  comparisons,  is 
satisfactory,  and  MODIA  cost  information  Is  within  +3%  of  the  comptroller's  figure. 

TA3LE  3 

COMPARISON  OF  MODIA  PREDICTIONS  WITH  ACTUAL  RESULTS 
(COURSE  3ABR32834) 


Data  Description 

MODIA 

Difference 

Annual  Student  Entry 

316 

316 

Annual  Graduates 

214 

214 

Average  Course  Hours 

510 

510 

Annual  Failures 

6 

6 

Resource  Utilization  (%) 

60.2 

62 

1.8 

Instructors 

28 

28 

Cost  Per  Student  Week 

$399.53 

$411.94 

$12.41 

Cost  Per  Graduate 

$6,792.01 

$7,002.98 

$210.97 

Total  Course  Cost  (1  Year) 

$1,453,490.10 

$1,498,637.70 

$45,147.60 

b.  Course  3ABR30434,  Ground  Radio  Equipment  Repairman:  This  is  a self-paced  instruc- 
tional system  comprised  of  420  instructional  hours  for  this  course.  Table  4 shows  results 
obtained  by  ISO  Team  B.  System  fidelity,  as  evidenced  by  data  comparisons.  Is  satisfactory 
and  MODIA  cost  data  are  within  +7.5%  of  the  comptroller's  figure  for  this  course. 


TABLE  4 

COMPARISON  OF  MODIA  PREDICTIONS  WITH  ACTUAL  RESULTS 
(COURSE  3ABR30434) 


Data  Description 

■KMiJffli 

Actual 

Difference 

Annual  Student  Entry 

746 

746 

- 

Annual  Graduates 

532 

534 

2 

Average  Course  Hours 

453.4 

420 

33.4 

Annual  Failures 

6 

4 

2 

Resource  Utilization  {%) 

65 

65 

- 

Instructors 

25 

25 

- 

Cost  Per  Student  Week 

$335.62 

$333.11 

$2.51 

Cost  Per  Graduate 

$5,034.30 

$4,663.54 

$370.76 

Total  Course  Cost  (1  Year) 

$2,678,247.60 

$2,490,330.30 

$187,917.30 

c.  Course  3ABR30331,  ATC  Radar  Repairman:  This  is  a lock  step  instructional  system 
comprised  of  690  instructional  hours.  Table  5 reveals  results  obtained  by  ISD  Team  C. 
System  fidelity,  as  indicated  by  data  comparisons,  is  satisfactory  and  MODIA  cost  data  are 
within  +2.9%  of  the  comptroller's  figure. 


TABLE  5 

COMPARISON  OF  MODIA  PREDICTIONS  WITH  ACTUAL  RESULTS 
(COURSE  3ABR30331 ) 


Data  Description 

Bn  'H  BBWWBI 

Annual  Student  Entry 

261 

261 

- 

Annual  Graduates 

242 

248 

6 

Average  Course  Hours 

691.5 

690 

1.5 

Annual  Failures 

4 

3 

1 

Resource  Utilization  (%) 

28 

30 

2 

Instructors 

23 

23 

- 

Cost  Per  Student  Week 

$434.98 

$413.45 

$21.53 

Cost  Per  Graduate 

$10,026.29 

$9,509.35 

$516.94 

Total  Course  Cost  (1  Year) 

$2,426,362.10 

$2,353,318.80 

$68,043.30 

d.  Course  3ABR30333,  Automatic  Tracking  Radar  Repairman:  The  instructional  design 
is  lock-step  with  a total  of  570  instructional  hours.  Table  6 reflects  results  obtained 
by  ISD  Team  D.  System  fidelity,  as  evidenced  by  data  comparisons,  is  satisfactory  and 
MODIA  cost  data  are  within  +5.5%  of  the  comptroller's  figure. 

TABLE  6 

COMPARISON  OF  MODIA  PREDICTIONS  WITH  ACTUAL  RESULTS 
(COURSE  3ABR30333) 


Data  Description 

MODIA 

Actual 

Annual  Student  Entry 

155 

155 

- 

Annual  Graduates 

96 

93 

3 

Average  Course  Hours 

576 

570 

6 

Annual  Failures 

7 

10 

3 

Resource  Utilization  (%) 

33 

34 

1 

Instructors 

14 

14 

- 

Cost  Per  Student  Week 

$374.80 

$413.45 

$38.65 

Cost  Per  Graduate 

$7,483.51 

$8,174.53 

$691.02 

Total  Course  Cost  (1  Year) 

$718,416.96 

$760,231.29 

$41,814.33 

1 


e.  Course  3AQR30020-1,  Electronics  Principles:  This  is  a modular/self-paced 
instructional  system  comprised  of  450  hours  of  instruction.  Due  to  the  laroe  student  flow, 
bulk  of  planning  data, and  the  contraints  in  the  Resource  Utilization  Model  (RUM)  program, 
the  MODI  A operators  and  course  planners  could  only  simulate  600  hours  of  course  operation. 
Thus,  MODIA  results  (annual  graduates,  average  time  to  graduate,  and  student  failures)  were 
estimated  for  one  year  based  on  the  results  for  the  600  hours  of  course  simulation.  However, 
the  MODIA  Cost  Model  (M0DC0M)  was  exercised  with  data  for  predicting  cost  for  one  year. 

This  was  possible  because  the  M0DC0M  can  be  operated  apart  from  the  MODIA  User  Interface 
(UI)  and  Resource  Utilization  Model  (RUM)  computer  programs.  Table  7 reflects  results 
obtained  by  ISD  Team  E. 

TABLE  7 

COMPARISON  OF  MODIA  PREDICTIONS  WITH  ACTUAL  RESULTS 
(COURSE  3AQR30020-1) 


Data  Description 

MODIA 

Actual 

Difference 

Annual  Student  Entry 

6,687 

6,687 

- 

Annual  Graduates 

5,980 

6,315 

335 

Average  Course  Hours 

273 

450 

177 

Annual  Failures 

493 

390 

103 

Resource  Utilization  (%) 

21.45 

35 

13.55 

Instructors 

194 

201 

7 

Cost  Per  Student  Week 

$375.89 

$310.25 

$65.64 

Cost  Per  Graduate 

$3,420.60 

$4,653.75 

$1,233.15 

Total  Course  Cost  (1  Year) 

$20,455,188.00 

$29,388,431.00 

$8,933,243.00 

The  large  difference  between  the  actual  course  cost  for  one  year  and  that  cost 
calculated  by  MODIA  (30.39%)  does  not  invalidate  the  MODIA  computer  programs.  Linear 
projection  of  a year's  operation  (1,500  hours)  based  on  600  hours  of  simulation  may  have 
been  inappropriate.  The  congruity  between  actual  and  projected  costs  for  the  other  four 
courses  substantiates  MODIA's  validity.  However,  the  system  limitation  experienced  with 
Course  3AQR30020-1  does  point  out  one  deficiency  with  the  current  programs  - the  inability 
to  handle  very  large,  complex  courses. 

11.  MODIA  System  Effectiveness:  The  effectiveness  of  the  MODIA  System  as  a planning  tool 
was  measured  in  two  ways.  First,  ISD  teams  were  acquainted  with  the  MODIA  System  by  requir- 
ing them  to  develop  the  current  designated  service  test  course  as  described  by  the  Plan 
of  Instruction  (POI).  Having  been  trained  how  to  operate  the  system  with  the  help  of  the 
MODIA  Interface  Team,  each  team  was  then  challenged  to  consider  instructional  alternatives 
which  might  produce  a more  cost-effective  course.  Each  ISD  team  developed  at  least  two 
alternatives.  The  effectiveness  of  MODIA  as  a planning  tool  was  further  pursued  by  asking 
ISD  team  members  specific  questions  designed  to  determine  the  system's  strengths  and 
weaknesses . 

a.  Instructional  Alternative  Development  by  ISD  Teams:  Having  developed  each  course 
through  application  of  Instructional  System  Development  (ISD)  methodology,  each  team  was 
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made  -ware  that  MODIA  was  designed  to  assist  planners  only  after  completion  of  Steps  1, 

2,  and  3 of  the  5-Step  ISD  Model: 

(1)  Analyze  system  requirements. 

(2)  Define  training  requirements. 

(3)  Develop  objectives  and  tests. 

(4)  Plan,  develop,  and  validate  instruction. 

(5)  Conduct  and  evaluate  instruction. 

MODIA  was  used  by  the  teams  to  complete  the  planning  phase  of  Step  4 by  aiding  them  in 
sequencing  learning  activities  and  selecting  instructional  methods  and  resources. 

b.  MODIA  Design  Process:  For  each  course,  team  members  used  the  following  procedures 
and  factors  to  plan  and  develop  instructional  alternatives  with  MODIA: 

(1)  Select  the  sequencing  of  teaching  objectives  and  assign  subject  matter 
types,  resources,  and  time  to  teach  the  objectives. 

(2)  Determine  where  in  the  course  any  tests,  reviews,  and  critiques  are  to  be 
administered  and  assign  special  resources  to  tests  and  reviews  plus  the  time  these 
activities  will  require. 

(3)  Determine  method(s)  by  which  students  are  taught. 

(4)  Determine  instructional  design. 

(5)  Determine  annual  student  input,  size  of  the  entering  group,  and  time  between 

entries. 

(6)  Specify  the  length  of  training  day,  section  size  (maximum  and  minimum  number 
of  students  in  the  group  instructed),  teaching  agent  (instructor  or  student),  and  teaching 
format  (simple,  recitation,  group-interactive  or  response-paced). 

(7)  Specify  test  details  by  designating  the  proportion  of  students  allowed  to 

fail . 

(8)  Identify  the  type  and  quantity  of  resources  needed  for  teaching  each 
learning  event. 

(9)  Specify  the  maximum  and  minimum  number  of  students  that  may  form  a section 
be  be  handled  by  each  unit  of  resources. 

In  planning  instructional  alternatives,  the  teams  were  allowed  to  vary  all  planning 
factors  except  the  annual  student  input  and  the  student  failure  rate.  This  was  decided 
because  course  planners  felt  that  these  were  two  factors  over  which  they  had  no  control. 
Thus,  the  annual  student  input  and  failure  rate  used  in  the  MODIA  design  process  were  the 
established  annual  Trained  Personnel  Requirement  (TPR)  and  failure  rate  respectively  for 
each  course  selected  for  the  service  test.  The  impact  of  alternative  course  designs 
considered  by  the  planners  are  presented  and  discussed  for  each  course. 

c.  Course  3ABR32834:  The  present  instructional  design  is  group/lock-step  with  510 
instructional  hours,  an  annual  entry  of  316  students  (entry  rate  of  10+2  students  every 
30  hours).  Further,  the  present  instructional  system  is  designed  for  each  instructor  to 
be  assigned  10+2  students  during  classroom  theory  and  5+1  students  during  equipment 
performance.  The  course  is  structured  so  that  theory  of  equipment  operation  is  presented 
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TABLE  8 

COMPARISON  OF  MODIA  PLANNED  ALTERNATIVE  AND  MANUAL  ALTERNATIVE  (COURSE  3ABR32834) 


Data  Description 


Course  Length  (Weeks) 
Student  Input  (1  Year) 
Annual  Graduates  (Average) 
Failures 

Current  Number  Students  in 
Course 

Average  Time  to  Graduate 
(Weeks) 

Instructors  (%  Utilization) 

Rooms  (%  Utilization) 

Hardware  (%  Utilization) 

Test  Equipment  (%  Utili- 
zation) 

Resources  Queue  (Average 
Hours) 

Queue  for,  Other  Students 
(Average  Hours) 

Cost  Per  Student  Week 

Cost  Per  Graduate 
(Average) 

Total  Course  Cost-  1st 
Year 

Total  Course  Cost  - 
5 Yr  Cum 


28  (62%) 
17  (75%) 
56  (35%) 
208  (39%) 


30  (66%) 
22  (66%) 
84  (29%) 
312  (29%) 


33  (59%) 
22  (90%) 
84  (80%) 
312  (80%) 


$518.77 

$9,120.00 


$1,951,700.00 


$7,063,600.00 


$1,584.00 

$23,682.24 


$5,068,000.00 


$10,473,000.00 


$1,668.68 

$22,043.29 


$5,092,000.00 


$10,394,200.00 


* Obtained  from  the  I&M  Unit  in  TCHTG  3380/TAVI. 


in  one  room,  additional  lecture  on  equipment  operation  is  presented  in  the  laboratory,  and 
unguided  practice  on  the  equipment  in  the  same  laboratory  is  supervised  by  one  instructor. 


When  team  memters  exercised  MOOIA  to  develop  baseline  data,  they  found  that  students 
were  queuing  for  equipment  and/or  instructors  due  to  two  or  more  student  groups  requiring 
use  of  these  resources  at  the  same  time.  Based  on  experience,  course  planners  felt  that 
guided  and  unguided  practice  learning  events  for  each  unit  of  hardware  could  be  combined 
and  the  time  for  accomplishing  these  events  decreased  (if  each  class  was  divided  Into  two 
equal  groups  with  one  instructor  assigned  per  group  during  performance  time). 


Alternative  1 was  created  with  a group/lock-step  instructional  design,  an  annual  entry 
of  316  students,  and  decreased  student  performance  time.  Here,  one  instructor  with  the 
capacity  to  teach  10+2  students  was  allocated  to  the  theory  classroom  and  two  instructors 
with  the  capacity  to  teach  10+2  students  were  allocated  to  the  equipment  laboratory  to 
accommodate  student  performance. 

Alternative  2 explored  a self-pace  approach  with  an  annual  entry  of  316  students  and 
a student  entry  rate  of  2+1  every  six  hours.  Further,  student  performance  time  was  kept 
the  same  as  for  Alternative  1 (.4-1.7  of  the  average  time  to  complete  a learning  event), 
each  theory  classroom  was  assigned  a capacity  of  10+2  students  with  one  instructor,  and 
each  equipment  room  was  assigned  a capacity  of  10+2  students  with  two  instructors. 

For  both  alternatives  plus  the  manually  planned  course,  the  MOOIA  System  projected 
resource  requirements  to  support  the  courses.  The  planning  strategy  used  by  the  ISD  teams 
was  to  determine  the  effects  the  above  changes  in  Alternative  1 and  Alternative  2 would  have 
on:  (1)  average  time  for  students  to  graduate,  (2)  quantity  of  resources  and  percentage 
resources  utilization,  (3)  student  queue  time  for  resources  and  other  students,  arid  (4) 
course  cost.  Table  8 depicts  data  as  a basis  for  comparing  the  MOOIA  and  manual  planning 
results. 

The  data  presented  in  Table  6 reveal  the  following:  (1)  Students  were  able  to 
graduate  approximately  three  and  four  weeks  earlier  in  Alternatives  1 and  2,  respectively, 
compared  to  the  present  course;  (2)  Additional  resources  were  required  in  both  MOOIA 
alternatives  compared  to  the  present  instructional  system;  however,  the  increase  in 
resources  eliminated  student  queuing  on  resources,  (3)  First  year  total  and  five  year 
cumulative  course  cost  is  considerably  higher  for  both  MODIA  alternatives  as  compared  to 
the  manually  planned  system,  and  (4)  Because  of  decreased  course  length,  both  Alternatives 
1 and  2 could  accommodate  more  students  per  year  than  the  existing  course. 

The  cost  data  indicate  that  neither  of  the  alternatives  chosen  for  simulation  surpassed 
the  existing  course  in  cost-effectiveness.  Reduction  in  course  length  and  increased 
resource  utilization  do  not  appear  to  be  worth  the  extra  cost.  This  does  not  imply  that 
a more  cost-effective  alternative  does  not  exist;  however,  using  what  were  considered  to 
be  logical  approaches  to  course  revision,  the  planners  were  unable  to  develop  less 
costly  alternatives  with  MODIA. 

d.  Course  3AI.R30434:  The  present  instructional  design  is  self-pace  with  a 
duration  of  420  instructional  hours,  an  annual  TPR  of  696  students,  and  a student  entry 
rate  of  2+1  students  every  hour.  In  the  present  course,  equipment  limitations  cause 
students  to  spend  an  extreme  amount  of  time  waiting  to  participate  in  hands-on  performance. 
Due  to  queuing  on  hardware,  it  takes  the  students  an  average  of  14.75  weeks  to  graduate. 
Taxed  with  an  annual  TPR  of  696  students,  course  planners  were  challenged  to  consider 
instructional  alternatives  which  might  alleviate  resource  queuing  and  at  the  same  time 
reduce  course  operating  cost. 
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Alternative  1 was  examined  with  a self-pace  instructional  design,  an  annual  TPR  of 
696  students,  and  a student  entry  rate  of  2+1  every  hour  with  MODIA  projecting  the 
resource  requirements  to  support  the  approach  to  prevent  student  queuing  on  resources. 

Alternative  2 explored  a group/lock-step  instructional  design  with  a student  entry 
rate  of  15  every  30  hours,  an  instructor  capacity  of  15  students  for  lecture/discussion, 
six  students  for  laboratory  performance  and  four  students  for  troubleshooting  and  aligning 
equipment,  and  a room  capacity  of  15  students  for  all  learning  events.  In  this  approach, 
the  time  allotted  for  student  achievement  of  learning  events  was  the  same  as  listed  in  the 
present  P01.  Also,  course  planners  allowed  MODIA  to  project  resource  requirements  to  support 
this  alternative.  Table  9 presents  data  as  a basis  for  comparing  MODIA  and  manual  planning 
approaches. 

TABLE  9 

COMPARISON  OF  MODIA  PLANNED  ALTERNATIVES  AND  MANUAL  ALTERNATIVE 
(COURSE  3ABR30434) 


MODIA 

Data  Description 

Manual 

Self-Pace 

nigrnji 

Course  Length  (Weeks) 

14 

14 

14 

Student  Input  (1  Year) 

*980 

*980 

*980 

Annual  Graduates  (Average) 

*696 

696 

711 

Failures 

*26 

26 

27 

Current  Number  Students  in 

Course 

258 

258 

242 

Average  Time  to  Graduate  (Weeks) 

14.75 

13.7 

14.58 

Instructors  (%  Utilization) 

22  (46%) 

18  (85%) 

30  (61%) 

Rooms  (%  Utilization) 

18  (70%) 

19  (62%) 

14  (73%) 

Hardware  (%  Utilization) 

9 (42%) 

18  (41%) 

16  (83%) 

Test  Equipment  (%  Utilization) 

1 (42%) 

1 (41%) 

1 (83%) 

Resources  Queue  (Average  Hours) 

52.8 

0 

0 

Queue  for  Other  Student 
(Average  Hours) 

0 

0 

7.8 

Cost  Per  Student  Week 

$318.34 

$338.88 

$451.29 

Cost  Per  Graduate  (Average) 

$4,698.70 

$4,642.67 

$6,579.95 

Total  Course  Cost  - 1st  Year 

$3,270,300.00 

$3, 231, 300  .'00 

$4,678,200.00 

Total  Course  Cost  - 5 

$9,218,600.00 

$9,170,500.00 

$10,708,300.00 

* Obtained  from  the  I&M  Unit  in  TCHTG  3410/TCSTI. 
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Results  indicate  that  students  in  both  the  MOOIA  alternatives  and  the  manual 
alternative  graduate  in  approximately  the  same  time,  but  Alternative  2 requires  an 
overall  increase  in  resources  (+8  instructors,  +7  pieces  of  hardware,  -4  classrooms). 

On  the  other  hand.  Alternative  1 calculated  by  MOOIA  requires  four  fewer  instructors  and 
only  one  additional  classroom  to  eliminate  the  long  wait  for  resources  and  graduate 
students  in  approximately  1.06  fewer  weeks.  These  data  indicate  that  students  matriculating 
in  MOOIA  Alternative  1 can  graduate  at  approximately  $65.03  less  than  in  the  present 
instructional  program.  Thus, it  appears  that  the  planners  for  Course  3ABR30434  were  able 
to  structure  a course  using  MODIA  which  is  more  cost-effective  than  the  current  instructional 
system. 

e.  Course  3A8R30331:  The  present  instructional  design  is  group/lock-step  with  690 
instructional  hours,  an  annual  TPR  of  140  students,  a student  entry  rate  of  11+1  students 
every  90  hours,  and  a failure  rate  of  three  percent.  In  the  present  course,  about  332 
hours  are  allotted  for  student  performance  with  the  remaining  358  hours  used  for  classroom 
theory.  Due  to  queuing  for  resources  and  other  students  (student  groups  having  to  wait 
until  resources  become  available  and  washbacks  having  to  wait  for  a class), it  takes  an 
average  of  24  weeks  for  students  to  graduate  in  the  present  course  as  calculated  with 
MODIA.  Further,  based  on  experience,  course  planners  felt  that  the  same  student 
proficiency  level  could  be  obtained  if  the  performance  time  was  reduced  by  one-half.  They 
were  also  of  the  opinion  that  queuing  for  resources  and  other  students  could  be  decreased 
if  the  quantity  of  resources  were  increased  and  the  student  capacity  each  could  handle 
was  decreased. 

Alternative  1 examined  a group/lock-step  design  with  an  annual  TPR  of  140  graduates, 
a student  entry  rate  of  11+1  students  every  90  hours,  and  a failure  rate  of  three  percent. 

The  planners  also  assigned  the  following  capacities:  (1)  Instructors:  11+1  students  for 
lecture/demonstration  and  5+1  for  student  performance,  (2)  Rooms:  11+1  students  each, 
and  (3)  Hardware  and  Associated  Test  Equipment:  5+1  students  each  during  performance 
time.  Further,  332  hours  and  358  hours  were  allotted  to  student  performance  and  classroom 
theory  respectively  with  MODIA  projecting  resource  requirements  to  support  the  approach. 

Alternative  2 explored  a group/lock-step  design  with  total  course  hours  reallocated 
so  that  181.5  hours  were  devoted  to  student  performance  and  508.5  hours  devoted  to  class- 
room instruction.  Student  input  and  the  number  of  students  assigned  to  each  resource  were 
the  same  as  for  Alternative  1,  and  MODIA  projected  resource  requirements  to  support  the 
approach.  Table  10  depicts  key  planning  results  as  a basis  for  comparing  MODIA  and  manual 
planning  effectiveness. 

The  figures  listed  in  Table  10  indicate  that  Alternative  1 graduates  students  in 
approximately  the  same  length  of  time  as  the  present  course,  reduces  queuing  time,  decreases 
percentage  of  resource  utilization,  and  increases  the  cost  to  graduate  the  140  students 
due  to  the  increase  in  the  quantity  of  resources  required  to  support  this  approach. 
Conversely,  Alternative  2 allows  students  to  graduate  on  an  average  of  3.9  hours  sooner  than 
the  present  course.  Further,  the  number  of  instructors  required  is  reduced  by  two  and  three 
additional  units  of  hardware  reduced  resource  queuing  to  zero  and  queuing  for  other  students 
by  IS  hours.  The  data  imply  that  the  increase  in  cost  for  the  additional  hardware  is  not 
offset  by  a decrease  in  both  the  number  of  instructors  and  the  average  time  to  graduate  and 
results  in  an  overall  increase  in  course  operation  cost  as  compared  to  the  present 
instructional  system  for  the  first  year.  However,  over  a five  year  period,  Alternative  2 
appears  to  be  the  better  course. 


TABLE  10 

COMPARISON  OF  MODIA  PLANNED  ALTERNATIVES  AND  MANUAL  PLANNED  ALTERNATIVE 

(COURSE  3ABR30331 ) 


MODIA 

Data  Description 

mM*JU,YUvi*m 

Course  Length  (Weeks) 

23 

23 

23 

Student  Input  (1  Year) 

*248 

*248 

*248 

Annual  Graduates  (Average) 

*140 

140 

140 

Failures 

*4 

4 

4 

Current  Number  Students  in 

Course 

104 

104 

104 

Average  Time  to  Graduate 
(Weeks) 

24 

24.67 

23.87 

Instructors  (%  Utilization) 

32  (45%) 

34  (36%) 

30  (32%) 

Rooms  (%  Utilization) 

8 (44%) 

8 (49%) 

8 (82%) 

Hardware  (%  Utilization) 

15  (39%) 

26  (25%) 

18  (29%) 

Test  Equipment  (%  Utilization) 

12  (39%) 

12  (25%) 

12  (29%) 

Resources  Queue  (Average  Hours) 

24 

0 

0 

Queue  for  Other  Students 
(Average  Hours) 

21 

7 

2 

Cost  Per  Student  Week 

$686.33 

$3,213,56 

$969.45 

Cost  Per  Graduate  (Average) 

$16,472.86 

$79,278.57 

$23,140.71 

Total  Course  Cost  - 1st  Year 

$2,306,200.00 

$11,099,000.00 

$3,239,700.00 

Total  Course  Cost  - 5 Yr  Cum 

$11,143,700.00 

$20,295,800.00 

$10,970,400.00 

* Obtained  from  the  I&M  Unit  in  TCHTG  3395/TE0TI. 

f.  Course  3ABR30333:  The  present  instructional  design  is  group/lock-step  with  570 
instructional  hours.  Planning  factors  in  Its  design  included  an  annual  TPR  of  96  graduates, 
a student  entry  rate  of  10+2  students  every  120  hours,  a failure  rate  of  4.5%,  two  units  of 
hardware  and  associated  test  equipment,  and  six  rooms.  Results  of  the  manually  planned 
instructional  alternative,  as  calculated  with  MODIA,  are  shown  in  Table  11. 

Holding  the  above  planning  factors  constant,  the  ISD  team  developed  Alternative  1 by 
decreasing  the  quantity  of  available  resources  and  assessing  the  impact  on  percentage  of 
resource  utilization,  queuing  time,  average  time  for  students  to  graduate,  the  number  of 
graduates,  and  the  course  operation  cost.  This  approach  was  used  because  of  the  low 
student  flow  and  course  planners'  belief  that  they  could  fulfill  the  annual  graduate 
requirment  with  less  hardware. 


Alternative  2 assessed  the  impact  on  Instructor  and  room  requirements  by  exploring 
a group/lock-step  approach  with  an  increase  ir  hardware  and  instructors  and  the  remaining 
planning  factors  held  constant.  Course  planners  used  this  approach  because  they  had  two 
additional  hardware  units  on  hand  and  wanted  to  determine  if  using  three  hardware  units 
would  decrease  the  length  of  the  course  sufficiently  to  offset  the  added  maintenance  and 
instructor  costs.  Table  11  presents  data  which  compares  the  MODIA  and  manual  planning 
results. 


TABLE  11 

COMPARISON  OF  MODIA  ALTERNATIVES  AND  MANUAL  PLANNED  ALTERNATIVE 
(COURSE  3ABR30333) 


■nm 

Data  Description 

Alternative  2 
Lock-Step 

Course  Length  (Weeks) 

19 

26 

17 

Student  Input  (1  Year) 

*155 

*155 

*155 

Annual  Graduates  (Average) 

*96 

98 

102 

Failures 

*7 

7 

7 

Current  Students  in  Course 

52 

70 

46 

Average  Time  to  Graduate 
(Weeks) 

19.77 

26.98 

17.48 

Instructors  (%  Utilization) 

14  (75%) 

12  (78%) 

18  (68%) 

Rooms  (%  Utilization) 

6 (53%) 

5 (60%) 

6 (47%) 

Hardware  (%  Utilization) 
MSQ-77 

2 (73%) 

1 (78%) 

3 (35%) 

Test  Equipment  (%  Utili- 
zation) 

2 (73%) 

2 (78%) 

2 (35%) 

Resources  Queue  (Average 
Hours) 

0 

0 

0 

Queue  for  Other  Students 
(Average  Hours) 

1.3 

4.1 

1.3 

Cost  Per  Student  Week 

$443.22 

$368.60 

$648.80 

Cost  Per  Graduate 
(Average) 

$8,762.50 

$9,944,90 

$11,328.43 

Total  Course  Cost  - 1st 

Year 

$841,200.00 

$775,702.68 

$1,155,500.00 

Total  Course  Cost  - 5 Yr 
Cum 

$1,804,600.00 

$1,738,602.68 

$2,113,900.00 

* Obtained  from  the  I&M  Unit  in  TCHTG  3395/TEOTI. 
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For  Alternative  1,  course  operation  cost  is  less  compared  to  the  present  course. 
However,  it  requires  students  longer  to  graduate  since  students  who  fail  a block  of 
instruction  must  wait  for  a class  to  form  before  retaking  the  subject  material.  Thus, 
the  data  imply  that  the  initial  savings  for  Alternative  1 would  hav->  been  greater  if  a 
si ightly  higher  student  entry  rate  could  have  been  postulated.  As  with  the  two  preceeding 
courses,  planners  used  MODIA  to  develop  an  instructional  alternative  which  appears  to  be 
more  cost-effective  than  the  present  course. 

In  Alternative  2,  student  graduation  time  was  less  than  the  present  course.  However, 
this  time  saving  was  offset  by  the  salaries  of  the  additional  instructors  and  the  cost  of 
maintaining  the  additional  hardware  used  to  build  this  alternative.  Therefore,  as 
compared  to  the  present  manual  course,  the  data  Indicate  it  would  initially  cost  more  to 
increase  instructor  and  hardware  requirements  to  graduate  students  an  average  of  2.29  weeks 
sooner. 


g.  Course  3AQR30020-1:  This  course  incorporates  a modular/self-pace  instructional 
design  comprised  of  450  instructional  hours.  The  present  instructional  program  was  planned 
with  a TPR  of  4,073  graduates,  a student  input  rate  of  12+11  students  every  six  hours,  and 
an  established  failure  rate  of  five  percent.  Also,  students  entering  this  course  are 
assigned  to  four  different  categories  representing  four  AFSC's.  Given  these  planning 
factors  and  the  remaining  data  per  present  POI,  MODIA  was  exercised  to  calculate  the  base- 
line data  which  was  then  used  to  compare  the  results  of  planning  instructional  alternatives 
with  MODIA.  Because  of  the  tremendous  quantity  of  planning  data,  large  student  flow,  and 
the  constraints  in  the  Resource  Utilization  Model  (RUM)  program,  team  members  could  only 
simulate  600  hours  of  course  operation  to  develop  the  baseline  data.  Thus,  the  results  of 
manual  planning  as  shown  in  Table  12  were  estimated  for  one  year  based  on  the  results  of 
600  hours  of  course  simulation.  However,  as  in  the  validation  phase,  course  planners  were 
able  to  use  the  present  POI  planning  data  and  operate  the  Cost  Model  (M0DC0M)  to  get  the 
cost  of  the  present  system  apart  from  course  operation  simulation. 

Faced  with  not  being  able  to  adequately  plan  and  simulate  this  course  with  the  present 
quantity  of  planning  data,  team  members  used  the  following  design  strategy  to  plan  one 
instructional  alternative  with  MODIA: 

(1)  Specified  student  input  at  5,908. 

(2)  Specified  entry  rate  of  7+2  students  every  six  hours. 

(3)  Used  tracking  for  course  diversification  in  which  students  are  divided  into 
four  categories  with  each  category  of  students  designated  to  skip  certain  learning  events 
and  take  learning  events  common  to  all  students. 

(4)  Packaged  learning  events  to  stay  within  the  250  limit. 

(5)  Used  the  same  instructor  ranks  (civilian  and  military)  as  present  course. 

(6)  Resequenced  learning  events  from  present  Plan  of  Instruction  (POI)  which 
allowed  students  to  take  comon  subject  master  first  then  branch  off  to  take  material 
common  to  their  AFSC. 

(7)  Allowed  MODIA  to  generate  resources  to  support  the  course  design. 
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In  generating  the  MODIA  alternative,  planners  attempted  to  eliminate  resource 
bottlenecks  in  an  effort  to  graduate  students  sooner  and  thus  produce  a cost  savings 
in  course  operation.  Table  12  shows  data  as  a basis  for  comparing  the  MODIA  and  manual 
planning  results. 

TABLE  12 

COMPARISON  OF  MODIA  AND  MANUAL  PLANNED  ALTERNATIVES 
(COURSE  3AQR30020-1 ) 


Course  Length  (Weeks) 

15 

15 

Student  Input  (1  Year) 

*5,908 

*5,908 

Annual  Graduates  (Average) 

*4,073 

**4,224 

Failures 

*301 

310 

Current  Number  Students  in  Course 

1,534 

1,354 

Average  Time  to  Graduate  (Weeks) 

12 

12.4 

Instructors  (%  Utilization) 

**178  (59%) 

**168  (50%) 

Hardware  and  Test  Equipment  (%  Utili- 
zation) 

**810  ( 5%) 

**819  ( 6%) 

Rooms  (%  Utilization) 

**65  (42%) 

**65  (44%) 

Resources  Queue  (Average  Hours) 

1.5 

0 

Queue  for  Other  Students  (Average  Hours) 

0 

0 

Cost  Per  Student  Week 

$312.50 

$293.75 

Cost  Per  Graduate  (Average) 

$4,062.44 

$3,642.46 

Total  Course  Cost  - 1st  Year 

$16,546,300.00 

$16,458,600.00 

Total  Course  Cost  - 5 Yr  Cum 

$56,865,900.00 

$52,362,100.00 

* Obtained  from  I&M  Unit  in  TCHTG  3395/TEOTI. 
**  Estimated  from  a 600-hour  RUM  simulation. 


Results  shown  in  Table  12  for  the  MODIA  alternative  are  estimated  from  600  hours  of 
course  simulation.  As  can  be  seen  from  Table  12,  students  take  .4  weeks  longer  to 
graduate  in  the  MODIA  alternative  as  estimated  from  the  220  who  graduated  during  the  600 
hours  simulation.  Although  this  alternative  indicates  a savings  in  the  number  of 
instructors  utilized,  the  total  savings  in  instructor  usage  is  offset  somewhat  by  an 
increase  in  the  number  of  hardware  units  required  to  decrease  resource  queue  time  to  zero. 
Finally,  the  Initial  and  five-year  course  operation  costs  for  the  MODIA  alternative  are 
less  than  the  present  course  costs;  however,  this  figure  is  based  on  estimates  and  is  not 
necessarily  reliable. 
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h.  Course  3AIR75100-X:  This  is  a new  course  which  was  developed  to  train  technical 
instructors  how  to  teach  in  a self-pace  environment.  The  Instructional  design  is  a self- 
pace/grpup-pace  combination  to  accompany  four  categories  of  students  (slow  group-pace, 
slow  self-pace,  fast  group-pace,  and  fast  self-pace).  Prior  to  entering  the  course, 
students  are  administered  a test  to  determine  to  which  category  they  will  be  assigned. 

Table  13  shows  the  results  of  Course  3AIR75100-X  as  designed  by  Lowry  course  planners  using 
the  MODIA  System.  Thefe  are  no  data  for  comparing  these  results  since  this  Is  a new  course. 
The  course  planners  designed  the  instructional  program  using  the  following  planning 
factors:  (1)  a TPR  of  529  graduates,  (2)  an  entry  rate  of  2+1  students  every  six  hours,  (3) 
a three  percent  failure  rate,  (4)  four  different  categories  of  students,  (5)  content 
diversification  for  each  category,  (6)  six  Instructors,  (7)  three  rooms,  and  (8)  three 
hardware  units. 

TABLE  13 

COURSE  3AIR75100-X , COURSE  DESIGN 


Data  Description 

MODIA  Output 

Sel f-Pace/Group-Pace 

Course  Length  (Weeks) 

4 

Student  Input  (1  Year) 

544 

Annual  Graduates  (Average) 

529 

Failures 

15 

Current  Number  Students  in  Course 

0 

Average  Time  to  Graduate  (Weeks) 

3.4 

Instructors  (%  Utilization) 

9 (27%) 

Rooms  (%  Utilization) 

3 (96%) 

Hardware  (%  Utilization) 

3 (16%) 

Resources  Queue  (Average  Hours) 

9 

Queue  for  Other  Students  (Average  Hours) 

5.33 

Cost  Per  Student  Week 

$508.79 

Cost  Per  Graduate  (Average) 

$1,729.87 

Total  Course  Cost  - 1st  Year 

$915,101.23 

Total  Course  Cost  - 5 Yr  Cum 

$4,225,400.00 

Four  different  alternatives  were  attempted  using  MODIA;  however,  Lowry  planners  were 
of  the  opinion  that  the  results  shown  in  Table  13  would  provide  an  optimum  instructional 
program  for  Course  3AIR75100-X.  The  average  time  for  all  students  will  be  3.4  weeks. 

The  slow  group-pace  students  and  slow  self-pace  students  will  graduate  In  3.93  and  3.02 
weeks  respectively.  The  fast  group-pace  students  and  the  fast  self-pace  students  will 
graduate  in  3.68  and  2.97  weeks  respectively.  Further,  student  queue  for  resources  and 
other  students  is  very  high.  The  waiting  for  resources  is  mainly  due  to  self-pace 
students  having  to  wait  to  gain  access  to  hardware.  One  specific  unit  of  hardv?are  which 
caused  the  majority  of  resource  queuing  was  a visual  trainer  used  with  the  computer  for 
particular  learning  events.  The  majority  of  student  queuing  is  for  student  washbacks  in 
the  group-pace  categories  having  to  wait  to  join  a class. 
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One  problem  encountered  by  Lowry  course  planners  was  that  they  were  unable  to  use 
MODIA  to  simulate  students  skipping  learning  events  due  to  resource  unavailability  and 
then  returning  to  those  events  when  the  resources  become  available.  During  actual 
course  operation,  this  flexible  scheduling  will  be  used,  and  the  Lowry  planners  believe 
actual  resource  queue  will  oe  nearly  zero. 

i.  Structured  Interview:  Another  means  to  determine  the  effectiveness  of  MODIA  as 
a planning  tool  was  to  assess  the  reactions  of  service  test  participants  (by  structured 
interview)  to  planning  instructional  alternatives  with  MODIA.  Personnel  involved  in  the 
service  test  were  able  to  achieve  a satisfactory  level  of  expertise  in  MODIA  and  discovered 
many  strengths  and  weaknesses  of  the  system  during  the  service  test.  At  the  completion  of 
the  evaluation  period,  members  of  the  MODIA  Interface  and  ISO  teams  were  individually 
interviewed  to  determine  their  perception  of  MODIA's  value  as  a planning  tool  and  to 
identify  factors  affecting  its  effectiveness  (see  Appendices  1 and  2).  Assessments  to 
team  member  responses  are  provided  below: 

(1)  MODIA  Value  as  a Planning  Tool:  Team  members  were  of  the  opinion  that  MODIA 
will  enable  users  to  accomplish  the  following: 

(a)  Accurately  plan  new  and  revised  courses.  ISO  team  members  believed 
that  the  courses  simulated  by  MODIA  were  accurate  or  valid  in  the  sense  that  the  computer- 
generated courses  had  a great  deal  of  fidelity  to  the  actual  courses.  With  this  accuracy, 
planners  could  have  confidence  in  the  products  generated  by  MODIA  during  course  planning. 

(b)  Rapidly  plan  instructional  alternatives  once  planning  data  have  been 
collected  and  worksheets  prepared.  It  required,  on  the  average,  three  days  of  intense 
over-the-shoulder  guidance  by  the  MODIA  Interface  Team  for  each  ISD  team  to  collect  and 
translate  course  planning  data  to  MODIA  parameters  for  system  operation.  However, 
collecting  planning  data,  preparing  worksheets,  and  planning  one  instructional  alternative 
were  all  accomplished  in  about  one-fourth  the  time  required  to  plan  one  instructional 
alternative  manually. 

(c)  Rapidly  compare  the  consequences  and  costs  of  alternative  decisions 
concerning  an  instructional  system.  Examples  of  this  procedure  are  identified  in 
paragraphs  11c  through  llg  above  where  team  members  planned  two  alternatives  for  Courses 
3ABR32834,  3ABR30434,  3ABR30331,  and  3ABR30333.  From  MODIA  printouts,  team  members  were 
able  to  identify  areas  of  a course  (especially  in  resource  utilization)  that  could  be 
improved.  Although  the  planners  had  to  use  their  own  judgment  to  devise  the  changes  that 
might  improve  the  course,  MODIA  rapidly  gave  the  planners  feedback  on  the  consequences  of 
these  changes.  Therefore,  in  an  operational  environment,  planners  could  again  and  again 
modify  the  simulated  course  until  an  optimum  mix  of  instructors,  materials,  equipment, 
facilities,  and  sequence  of  learning  events  were  found  given  the  entry  rate  and  ability 
level  of  the  student  population. 

(d)  Systematical ly  design  a course  by  providing  guidance  to  planners  which 
assists  them  in  making  the  major  course  design  decisions.  Planners  felt  that  this  is 
accomplished  by  the  MODIA  technique  of  using  highly  detailed, branching  questionnaires  to 
guide  planners  in  considering  school  policies,  teaching  techniques,  resource  constraints, 
and  training  content  in  the  instructional  design  process. 

(e)  Consider  many  options  in  planning  instructional  alternatives.  Examples 
of  this  are:  (1)  content  diversification  where  different  students  are  Instructed  in 
different  skills  and/or  knowledges,  (2)  tracking  where  students  are  divided  into  two  or 
more  groups  that  are  instructed  separately,  and  (3)  grouping  where  trainees  are  separated 
into  groups  on  the  basis  of  some  important  characteristic. 
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(2)  MODIA  Effectiveness  Limitations:  The  MODIA  Interface  Team  and  ISD  teams 
experienced  some  difficulties  which  they  believe  hindered  the  effective  use  of  MODIA  in 
planning  Instructional  alternatives.  Based  on  their  findings  during  the  service  test, 
team  members  reached  the  following  conclusions: 

(a)  Due  to  built-in  program  limitations  and  computer  core  requirements, 

MODIA  cannot  be  used  effectively  to  plan  a very  large  course  with  many  planning  factors. 

For  example,  Course  3AQR30020-1  team  members  had  to  package  resources  to  keep  the  total 
resources  within  the  program  limitation  (31  resources).  During  course  operation  simulation, 
the  computer  failed  to  keep  track  of  all  students;  also.it  required  an  extreme  amount  of 
time  (approximately  1.5  hours)  to  simulate  600  hours  of  course  operation.  Thus,  team 
members  estimate  that  it  will  require  in  excess  of  three  hours  to  simulate  one  year's 
course  operation  for  any  course  similar  to  Course  3AQR30020-1.  Further,  in  its  present 
state,  MODIA  requires  that  approximately  70k  to  90k  bytes  of  sequential  file  storage  be 
available  before  course  operation  simulation  begins.  The  system  was  evaluated  using  the 
H-6060  computer  where  a total  of  110k  bytes  had  been  dedicated  to  time-sharing  operations. 
Thus,  MODIA  planners  were  competing  with  ongoing  computer  training  courses  for  this 

file  storage  and  experienced  long  delays  for  the  computer  to  become  available. 

(b)  MODIA  terminology  is  not  consistent  with  ISD  terms  and  was  confusing 
to  course  planners.  Some  MODIA  terms  which  caused  confusion  are  listed  below: 

1_.  Presentation/demonstration  and  group  discussion  are  called 
"learning  event  types"  in  MODIA;  whereas,  ISD  terminology  uses  types  of  instruction. 

2.  Teaching  format  determines  the  extent  to  which  the  instruction  is 
explicitly  structured  to  student  needs.  Team  members  had  difficulty  in  determining  what 
teaching  format  to  select  for  specific  learning  events  because  the  definitions  in  the 
MODIA  literature  are  not  sufficiently  explicit  to  aid  in  selecting  the  most  appropriate 
teaching  format  for  their  subject  content. 

(c)  A well  trained,  highly  motivated  MODIA  team  is  needed  as  an  interface 
between  the  MODIA  system  and  course  planners.  MODIA  is  out  of  the  ordinary  because  it 
requires  what  is  now  an  unfamiliar  approach  to  the  planning  process.  Thus,  team  members 
feel  that  for  some  time  people  will  have  to  be  trained  to  use  the  system  because  they  will 
first  need  training  to  understand  its  purpose.  Further,  because  MODIA  is  fairly  complex, 
expertise  is  required  to  maximize  system  effectiveness.  One  operating  concept  recommended 
by  participants  was  to  establish  a team  of  three  or  more  people  at  each  center  who  are  both 
expert  in  the  operation  of  MODIA  and  familiar  with  course  planning  and  school  policies. 

This  team  could  work  with  ISD  teams  and  plan  a revision/new  course  when  required. 

(d)  The  occasional  garbling  of  resources  when  allocating  resources  to 
learning  events  in  the  resource  constraints  phase  of  course  design  prevented  team  members 
from  assigning  specific  numbers  to  designate  quantity  of  resources  allocated.  Thus  many 
times  course  planners  had  to  alter  their  resources  so  that  the  computer  would  accept  a 
number.  For  example,  when  planning  Course  3ABR30434,  Alternative  2 (group/lock-step),  team 
members  could  not  assign  18  resources  to  learning  events  because  this  became  scrambled. 
However,  they  added  one  more  classroom  and  entered  19  resources  and  the  resource  assign- 
ment was  printed  out  satisfactorily.  Course  planners  feel  that  this  is  a serious  system 
deficiency. 
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(e)  The  Inability  to  backtrack  to  a point  within  a phase  of  the  design 
process  without  signing  off  from  the  computer  causes  instructional  planning  delays. 
Because  of  time-sharing,  planners  were  often  unable  to  get  back  on-line  without  a 
substantial  delay.  Changes  recommended  by  team  members  to  remove  this  limitation  plus 
other  delays  are  listed  below: 


1_.  Assignment  of  special  resources  could  be  moved  from  the  "I  Phase" 
of  the  User  Interface  (UI)  program  and  assigned  in  the  "R  Phase"  (Describe  Resources) 
during  course  design.  As  MODIA  is  now  developed,  resources  are  first  assigned  by  name 
to  learning  events  (I  Phase)  then  later  by  quantity  (R  Phase).  Team  members  believe  that 
a tremendous  amount  of  time  could  be  saved  in  designing  instructional  alternatives  if  both 
assignment  and  quantity  of  resources  were  designated  simultaneously  for  each  learning  event. 

2.  Revise  "P  Phase"  (where  teaching  policies  are  described)  to  add  a 
correction  option  at  the  final  listing  of  learning  events.  At  present,  to  make  a change 
in  teaching  policies,  users  must  sign  off  the  computer  and  sign  back  on  and  ask  for  the 
MODIA  program  to  begin  at  the  "P  Phase"  of  the  design  process.  As  noted  above,  signing 
back  on-line  to  make  a change  could  take  from  5-15  minutes. 

3.  Revise  "C  Phase"  (summary  of  resource  constraints)  to  make  optional 
the  receipt  of  a hard  copy  printout  of  learning  event  completion  times  and  group  section 
size  assignment.  The  printout  is  merely  a reproduction  of  what  planners  typed  for  entry 
into  the  computer.  Many  times  team  members  did  not  desire  to  see  a printout  which  included 
some  150  to  250  learning  events  which  required  an  extreme  amount  of  time.  At  present, 

a summary  of  resource  constraints  is  automatically  printed  at  the  remote  terminal  whether 
users  require  it  or  not. 

(f)  The  "RUM  Phase"  (Describe  RUM  Parameters)  should  be  changed  to  give 
course  planners  the  option  to  start  course  operation  simulation  with  students  in  the 
course  (course  revision)  or  with  the  first  student/student  group  that  enters  (new  course). 
This  would  enable  planners  to  get  a more  accurate  representation  of  resource  utilization 
for  an  instructional  alternative  for  a revised  course.  At  present,  the  Resource 
Utilization  Model  (RUM)  simulates  course  operation  as  if  each  course  is  a new  program  (i.e., 
with  the  first  student/student  group  that  enters). 

12.  MODIA  System  Acceptance:  To  measure  course  planner  acceptance  of  the  MODIA  system, 
replies  of  ISD  team  members  to  specific  questions  (Appendix  2,  Questions  12,18,23,28, 

36,38,  and  42)  were  collapsed  into  positive  and  negative  responses  with  positive 
responses  Indicating  acceptance  of  various  phases  of  MODIA.  The  frequency  of  these 
responses  by  each  ISD  team  is  depicted  in  Table  14. 

TABLE  14 

FREQUENCIES  WITH  WHICH  ISD  TEAM  MEMBERS  ACCEPTED  MODIA  AS  A PLANNING  TOOL 


Categories 


Positive 

Negative 


9 

8 

23 

19 

6 

5 

Most  teams  felt  positively  about  MODIA  and  would  accept  the  system  as  a planning  tool 
in  a technical  training  environment.  However,  significant  difference  in  opinions  does 
exist  (X‘=  34.71,  P£  .001).  During  the  structured  Interviews,  members  of  ISD  Team  D 
expressed  the  opinion  that  M0D1A  was  of  limited  utility  to  a course  planner.  They  felt 
that  at  their  level,  they  only  get  Involved  with  planning  a POI,  not  determining  course 
operating  cost,  available  facilities,  pieces  of  equipment.  Further,  the  two 
members  of  ISD  Team  E --the  team  which  was  unable  to  simulate  the  large  3AQR30020-1 
course  - felt  neither  strongly  positive  nor  negative  about  MODIA.  However,  the  four 
remaining  teams  clearly  supported  the  MOOIA  System. 

13.  MODIA  System  Usefulness:  The  usefulness  of  the  MODIA  System  as  a course  planning 
tool  is  a function  of  the  recurring  requirement  to  plan  new  courses  and  revise  existing 
ones.  At  Keesler,  this  is  represented  by  Types  2,3,  and  4 course  planning  efforts.  Type 
1 courses  are  formal  training  contracted  with  civilian  industrial  or  educational 
institutions;  Type  5 courses  are  special  or  regular  formal  training  conducted  for  the 
USAF  by  the  Army,  Navy,  DOD,  other  Government  agencies,  or  Air  Force  agencies  other  than 
ATC.and  Type  6 courses  are  regular  training  conducted  by  civilian  educational  institutions; 
therefore,  the  instructional  programs  for  Types  1,5,  and  6 courses  normally  are  not  planned 
by  course  planners  at  Air  Training  Command  Centers. 

In  order  to  obtain  an  estimate  of  the  annual  planning  requirement,  Keesler  USAFSAAS 
Plans  Branch  files  were  reviewed  to  determine,  by  type,  how  many  courses  were  in 
existence,  how  many  new  courses  were  implemented,  and  how  many  were  revised  during  calendar 
years  1970-1975.  Existence  of  training  plans  and/or  new  Plans  of  Instruction  (POI's) 
served  as  criteria  for  identifying  relatively  major  planning  efforts  only.  Appendices 
10  and  11  present  these  data  plus  the  number  of  course  weeks  revised  as  well.  Table  15 
provides  a summary  of  course  planning  efforts  by  type  during  the  period  1970-1975. 


TABLE  15 

KEESLER  SAAS  TYPES  2,3,  AND  4 COURSE  PLANNING  DURING  THE  PERIOD  1970-I97<; 


1 Type  2 1 

1 Type  3 

1 type  4 

1 No . Coursesl 

' 

I No.  Coursesl 

| 

[No.  Coursesl 

Year 

■•wnSSR 

i:rr» 

IfuTM 

Percent 

Rev 

Percent 

Rev 

111 

Percent 

1970 

94 

7 

9 

17% 

195 

57 

4 

31% 

33 

13 

11 

72% 

1971 

99 

13 

5 

18% 

200 

71 

5 

38% 

53 

21 

16 

70% 

1972 

105 

17 

6 

22% 

208 

102 

8 

53% 

69 

34 

30 

93% 

1973 

112 

46 

D 

47% 

216 

151 

8 

74% 

106 

41 

37 

74% 

1974 

122 

IP 

H 

23% 

243 

97 

27 

51% 

129 

25 

23 

37% 

1975 

126 

8 

10% 

247 

43 

4 

19% 

179 

67 

50 

65% 

Avg 

110 

18 

D 

22.8% 

218 

89 

9 

44.3% 

95 

33 
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Although  the  percentages  fluctuate  widely  from  year  to  year,  the  table  reveals  that 
on  the  average  23  percent  of  the  Type  2,  44  percent  of  the  Tyne  3,  and  69  percent  of  the 
Type  4 courses  are  either  revised  or  newly  implemented  each  year  at  Keesler. 


To  better  grasp  the  magnitude  of  the  course  planning  effort.  Table  16  depicts  the 
actual  number  of  course  days  either  newly  planned  or  revised  by  the  type  for  the  same  six 
year  period  (see  page  26  for  Table  16). 
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With  these  data,  one  can  calculate  the  percentage  of  course  days  revised  or  newly 
implemented  for  each  year.  For  instance,  during  1975,  24  percent  of  the  total  Types  2,3, 
and  4 course  days  In  existence  were  either  revised  or  newly  planned  and  implemented.  How- 
ever, these  figures  fluctuate  greatly  from  year  to  year.  During  1974,  the  corresponding 
percentage  was  52  percent.  To  make  this  figure  more  reliable,  six-year  averages  were  used, 
and  it  was  found  that  for  any  one  year  Keesler  is  likely  to  be  involved  in  planning 
approximately  47  percent  of  the  total  number  of  days  of  instruction  for  all  Types  2,3,  and 
4 courses.  However,  out  of  this  total  planning  effort,  only  11  percent  is  allocated  to 
Type  4 courses  and  nine  percent  to  Type  2 courses.  Thus,  fully  80  percent  of  Keesler's 
planning  effort  is  devoted  to  revising  and  implementing  Type  3 courses. 

For  each  Type  2,3,  and  4 course,  course  planners  use  the  planning  factors  needed  to 
construct  an  instructional  program  within  ATC  policies  and  regulations.  However,  when 
planning  for  Type  4 courses,  course  planners  are  able  to  use  up  to  two-thirds  of  existing 
Plans  of  Instruction  prepared  for  Type  2 courses.  This  results  in  a reduction  in  the  amount 
of  major  planning  effort  for  new  Type  4 courses.  Further,  Type  4 courses  are  revised  for 
only  two  reasons:  (1)  when  an  evaluation  reveals  the  appropriate  level  of  training  is  not 
being  conducted,  and  (2)  when  reimplemented  from  standby  status  after  a modification  in 
the  equipment  taught.  Type  2 courses  require  major  planning  when  being  implemented.  How- 
ever, revisions  are  minor  unless  a major  equipment  modification  is  made  during  the  training 
which  requires  a major  POI  change.  Also,  the  general  procedure  for  Type  2 courses  is  that 
training  will  be  incorporated  in  Type  3 basic  or  supplemental  courses  once  the  special 
training  is  completed.  On  the  other  hand.  Type  3 courses  are  normally  long  (average  about 
66  course-days  per  course),  high-flow  courses  which  require  major  planning  efforts  when 
implemented  or  when  an  evaluation  shows  that  a major  revision  is  required. 

Thus,  it  appears  that  the  MODIA  System  has  limited  applicability  to  Types  2 ar.d  4 
courses.  However,  results  cited  earlier  indicated  that  MODIA  can  be  used  effectively  to 
plan  Type  3 courses.  Since  approximately  80  percent  of  all  the  days  of  instruction  which 
are  planned  or  revised  at  Keesler  each  year  belong  to  Type  3 courses,  one  can  conclude 
that  MODiA  is  useful  in  this  training  environment. 


SECTION  D - DISCUSSION 


14.  General:  In  most  respects,  the  results  of  this  service  test  support  the  MODIA  System. 

From  the  comparative  data  plus  the  comments  of  the  ISD  teams  and  the  MODIA  Interface  Team, 
it  is  evident  that  MODIA  provides  a valid,  thorough,  and  systematic  approach  to  course 
planning.  Conversely,  the  programs  in  their  current  state  do  have  several  deficiencies, 
some  serious  enough  to  limit  the  general  applicability  of  the  system  (see  para  lli).  How- 
ever, the  acid  test  for  MODIA  revolves  around  the  answer  to  one  question:  Will  the  increased 
cost  of  the  computerized  planning  system  be  offset  by  more  cost-effective  technical  train- 
ing courses? 

15.  MODIA  System  Cost:  The  cost  differential  between  the  MODIA  approach  and  the  current 
planning  system  was  calculated  from  data  collected  during  the  service  test.  To  begin  with, 
this  center  experienced  a one-time  MOOIA  implementation  cost  of  $36,080.  This  total  is  the 
sum  of  personnel  and  hardware  costs  associated  with  installing/debugging  the  programs, 
training  a team  of  MODIA  experts  (the  Interface  Team),  and  shaking  down  the  system  on  the 
Honeywell  computer  (see  Appendix  6 for  costs  attributed  to  personnel  and  hardware  categories). 
With  the  experience  gained  in  this  effort,  Rand  could  probably  accomplish  these  tasks  with 
less  time  and  difficulty  at  other  centers.  However,  because  of  the  special  conditions 
associated  with  this  service  test  - special  arrangements  for  computer  access  and  personnel 
assignment  - this  cost  may  not  be  representative. 
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For  the  five  Type  3 courses  examined  during  the  service  test,  evaluators  determined 
that  the  average  cost  per  course  hour  for  computerized  planning  of  two  complete  instructional 
alternatives  was  $17.65  (see  Appendix  8).  Since  no  Type  2 or  4 courses  were  revised  during 
the  service  test,  no  cost  figure  was  available  for  this  type  of  planning.  Therefore, 
determination  of  MODIA  planning  cost  was  limited  to  Type  3 courses  only.  Table  16  reveals 
that  on  the  average,  7,083  Type  3 course  days  are  revised  and  implemented  each  year.  This 
figure  equates  to  42,498  course  hours.  Multiplying  per  hour  cost  by  the  number  of  hours 
yields  a maximum  theoretical  annual  MODIA  planning  cost  for  Type  3 courses  of  approximately 
$750,000.  When  the $36, 000  implementation  cost  is  added  plus  the  additional  salary  required 
to  support  a two-person  iGS-12,  E-7)  Interface  Team  for  one  year,  the  total  is  approximately 
$802,000. 

As  with  all  projected  cost  figures  reported  in  this  section,  this  cost  can  be  considered 
only  the  roughest  of  estimates.  To  begin  with,  it  is  based  on  data  collected  from  but  five 
courses,  a small  and  possibly  unreliable  sample.  Second,  the  per  hour  cost  of  $17.65  is 
accurate  for  the  service  test,  but  planners  were  simulating  complete  revisions  and  exper- 
imenting with  the  system.  These  activities  would  not  be  representative  of  normal  revision 
requirements  once  such  a novel  system  had  become  familiar.  Finally,  as  mentioned  above, 
the  implementation  cost  might  not  be  representative. 

From  this  MODIA  planning  cost  must  be  subtracted  the  cost  of  manual  planning  or  the 
cost  of  the  current  planning  methodology.  This  cost  is  almost  entirely  personnel  cost. 
Unfortunately,  no  accountability  system  exists  to  keep  track  of  the  manhours  expended  in 
Step  4 of  the  ISO  process.  Therefore,  the  manhours  devoted  to  this  effort  and  the 
corresponding  personnel  cost  had  to  be  estimated.  As  mentioned  previously  in  this  report, 
the  manual  cost  of  planning  was  calculated  at  $7.33  per  course  hour.  This  figure  obviously 
suffers  from  questionable  reliability.  Nonetheless,  it  serves  to  illustrate  that  the 
personnel  costs  of  the  current  system  nearly  equal  the  total  MODIA  cost  for  planning  one 
complete  revision  ($8.33,  see  Appendix  8),  including  charges  for  computer  usage. 

Again,  multiplying  per  hour  cost  by  42,498,  the  average  number  of  Type  3 course  hours 
revised  and  implemented  annually,  yields  a figure  of  $311,510.  Subtracting  this  cost  from 
the  MODIA  cost  produced  a $490,490  differential  in  favor  of  the  current  system  for  the 
first  year  of  operation. 

Several  comments  need  to  be  made  with  respect  to  this  cost  difference.  First,  it  is 
only  as  accurate  as  the  manual  manhour  estimates.  The  evaluators,  based  on  their 
experience  in  a technical  training  environment,  believe  these  estimates  are  reasonable  but 
certainly  there  is  a considerable  margin  for  error.  Second,  the  MODIA  planning  cost  is 
based  on  construction  of  two  complete  instructional  alternatives  whereas  the  manual  cost 
is  based  on  one  conventional  course  revision.  This  is  because  under  the  current  system, 
the  perceived  single  best  course  design  is  pursued  from  the  beginning.  Minor  modifications 
as  necessary  are  made  along  the  way.  This  is  somewhat  analogous  to  generating  one  alternative 
with  MODIA,  but  the  product  is  the  course,  not  a design  to  be  compared  to  other  designs. 

Since  the  major  purpose  of  MODIA  is  to  stimulate  the  creation  of  alternatives,  the  cost  to 
produce  two  alternatives  is  presented  here.  Although  no  data  were  gathered  to  support  this 
point,  it  is  likely  that  the  first  MODIA  alternative  will  take  the  most  personnel  and 
computer  time  and  thus  be  the  most  expensive.  As  the  critical  areas  of  the  simulated 
course  are  identified,  it  should  become  progressively  easier  and  quicker  to  make  changes 
which  will  improve  the  course.  Finally,  the  total  MODIA  planning  cost  for  one  year 
($802,000)  represents  the  cost  of  very  extensive  planning.  This  is  because  the  per  hour 
cost  of  $17.65  was  gathered  during  a period  when  planners  were  challenged  to  experiment 
with  innovations  to  completely  redesign  their  courses.  The  manual  planning  cost  per  hour, 
on  the  other  hand,  represents  varying  degrees  of  revision  but  almost  always  less  radical 
than  the  MODIA  approach.  All  of  these  factors  taken  together  tend  to  explain  some  of  the 
great  difference  between  the  MODIA  and  manual  planning  cost. 
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16.  MODIA  Effectiveness: 


a.  One  measure  of  planning  effectiveness  is  instructional  system  cost.  As  reported 
in  the  Results  Section  of  this  report,  planners  used  MODIA  to  revise  three  of  the  five 
existing  Type  3 courses  to  produce  a theoretical  cost  savings.  The  magnitude  of  these 
savings  differed  widely.  Course  3ABR30434,  a self-pace  program  of  420  instructional  hours 
was  revised  resulting  in  a five  year  savings  of  $48,100.  On  the  other  hand,  3ABR30331, 

a group/lock-step  course  of  690  instructional  hours  was  revised  and  produced  a savings  of 
$173,300  over  a five  year  period. 

With  such  a limited  sample  plus  the  myriad  of  factors  which  determine  the  cost  savings 
possible  from  any  given  course  revision,  it  is  impossible  to  project  savings  in  instructional 
system  cost  through  the  use  of  MODIA.  Certainly  the  service  test  courses  have  demonstrated 
that  cost  reductions  are  possible  and  that  the  magnitude  of  these  reductions  for  all  Type 
3 courses  may  be  formidable.  These  cost  figures  must  be  interpreted  with  caution;  however. 
Course  3ABR30331  has  not  been  revised,  implemented,  and  operated  for  five  years  at 
$173,000  less  than  the  ongoing  instructional  system.  Rather,  when  planning  factors  for  both 
the  existing  course  and  the  MODIA  alternative  were  manipulated  by  the  Cost  Model,  the 
alternative,  which  teaches  the  same  subject  matter  to  the  same  level  of  proficiency  to  the 
same  number  of  students,  was  found  to  be  less  expensive.  If  the  MODIA  process  is  valid, 
and  evidence  indicates  that  it  is,  it  follows  that  this  cost  difference  is  valid. 

Another  point  should  be  made  concerning  the  cost  differential  between  the  ongoing 
courses  and  the  MODIA  alternatives.  Where  the  MODIA  alternatives  were  considerably  more 
expensive  than  the  current  system,  such  as  in  Course  3ABR32834,  the  increased  cost  was  due 
primarily  to  the  procurement  and  maintenance  of  additional  pieces  of  hardware  and  test 
equipment.  For  local  training  managers,  trainer  procurement  cost  is  very  real  and  tangible. 
For  two  of  the  three  MODIA  alternatives  which  were  shown  to  be  less  expensive  than  the 
current  courses,  the  savings  were  realized  from  slightly  shorter  course  length  and  decreased 
instructor  manning.  Shorter  course  length  decreases  student  pay,  allowances,  and  support 
costs  attributed  to  the  course.  Decreased  instructor  manning  levels  decrease  course  cost 
in  the  same  manner.  These  savings,  while  real  in  terms  of  the  total  Air  Force  cost  for 
technical  training,  are  considerably  less  tangible  and  possibly  less  meaningful  to  local 
planners.  To  use  an  extreme  but  illustrative  example,  Keesler  is  now  conducting  a 6-hour 
Mission  Application  Seminar  program  for  all  non-prior  service  airmen  who  attend  training 
programs  at  this  center.  During  a year's  time,  approximately  17,000  airmen  will  complete 
this  training.  Using  an  E-l  salary  figure  of  $3.00  per  hour,  this  program  will  cost 
$306,000  per  year  in  student  salaries  alone  (17,000  X 6 X $3.00).  Reducing  this  course  to 
five  hours  would  reduce  the  student  salary  contribution  to  annual  course  cost  to 
$255,000  (17,000  X 5 X $3.00).  Obviously,  this  $51,000  "savings"  has  little  meaning  in 
this  context.  However,  if  the  one  hour  reduction  did  reduce  the  required  number  of 
classrooms,  instructors,  media  devices,  and  so  forth,  then  meaningful  and  more  tangible 
savings  would  be  realized.  Thus  the  nature  of  cost  savings  for  various  alternatives 
must  be  investigated.  Also,  it  is  possible  that  incentives  to  plan  courses  which  decrease 
costs  in  these  less  tangible  areas  might  be  less  powerful  than  incentives  to  cut  the 
inventory  of  expensive  trainers. 

b.  Effectiveness  of  a planning  system  can  be  assessed  in  other  areas  as  well.  Can 
use  of  the  system  produce  course  designs  which  will  operate  under  budget  ceilings,  or 
graduate  students  in  a fixed  period  of  time,  or  insure  little  wait  for  training  resources? 
Evidence  from  the  service  test  indicates  that  courses  can  be  designed  to  meet  criteria  in 
these  dimensions  of  course  operation.  Planners  were  very  effective  in  designing 
alternatives  that  eliminated  student  queuing  for  resources.  Given  other  criteria  such  as 
the  TPR  and  an  established  failure  rate  plus  a constraint  on  the  duration  of  the  course, 
the  evidence  indicates  that  an  alternative  could  be  designed  to  meet  the  stated 
requirements . 
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Although  courses  can  be  designed  to  operate  In  any  manner  desired,  cost  is  always 
a salient  factor.  Can  courses  be  designed  to  meet  operating  criteria  and  still  be 
realistic  in  cost?  Again,  evidence  from  the  service  test  indicates  MODIA  can  lead  to 
optimum  course  designs  which  balance  cost  against  the  criteria.  For  instance.  Alternative 
2 generated  for  Course  3ABR30331  reduced  the  wait  for  resources  from  24  hours  to  0 and  the 
wait  for  other  students  from  21  hours  to  2 and  still  was  projected  to  cost  approximately 
$173,000  less  over  a five  year  period. 

Of  course,  MODIA  did  not  design  the  optimum  configuration  for  3ABR30331.  Planners  did. 
Using  experience  and  logic,  planners  put  together  an  instructional  system  they  thought 
would  be  effective.  MODIA  manipulated  the  planning  factors  and  produced  a simulated  course 
and  its  associated  cost.  Based  on  these  findings,  a second  instructional  system  was 
designed,  simulated,  and  priced.  Selecting  an  optimum  instructional  design  would  require 
the  evaluation  of  both  alternatives  to  determine  the  one  which  would  meet  the  needs  of 
the  planners,  be  it  lowest  cost,  greatest  student  capacity,  shortest  wait  for  instructors, 
or  a combination  of  all  factors.  Thus,  the  effectiveness  of  MODIA  in  constructing  better 
courses  is  both  maximized  and  limited  by  the  talent,  ingenuity,  motivation,  and  analytical 
ability  of  the  MODIA  Interface  Team  and  the  course  planners. 

c.  To  return  to  the  original  question  in  this  section,  "Will  the  increased  cost  of 
the  computerized  planning  system  be  offset  by  more  cost-effective  technical  training?," 

a tentative  answer  can  now  be  given.  Including  the  cost  for  computer  usage,  MODIA  is 
definitely  more  expensive  than  the  current  planning  system.  However,  the  magnitude  of 
the  cost  difference  cannot  be  assessed  reliably  due  to  the  method  of  collecting  the 
manual  planning  cost  data.  From  the  small  sample  of  courses  examined  during  the  service 
test,  it  is  evident  that  planners  were  able  to  generate  alternatives  with  MODIA  which 
were  both  less  costly  and  more  effective  in  several  operating  dimensions.  Extrapolating 
to  the  large  number  of  Type  3 courses  revised  and  implemented  annually,  one  can  conclude 
that  MODIA's  cost  could  be  offset  by  the  more  cost-effective  courses  which  should  result. 
However,  it  should  be  stressed  that  the  projected  cost  reductions  may  occur  in  many  areas 
of  course  operation,  and  the  utility  of  these  reductions  would  have  to  be  examined  with 
respect  to  other  course  operating  criteria. 

d.  Another  indication  of  MODIA  planning  effectiveness  is  the  number  of  manhours  required 
to  support  a given  planning  effort.  As  depicted  in  Appendices  4,  5,7,  and  8,  the  number  of 
manhours  allocated  to  MODIA  planning  during  the  service  test  was  considerably  lower  than  the 
estimated  number  of  hours  required  to  do  the  same  planning  under  the  conventional  approach. 
Depending  upon  the  procedures  and  organizational  configuration  established  to  support  MODIA, 
these  manhour  savings  could  be  quite  large.  However,  the  service  test  did  not  attempt  to 
establish  necessary  procedures  or  configurations,  and  the  potential  savings  in  this  area  are 
thus  unknown. 

17.  MODIA  System  Limitations: 

a.  Although  the  products  of  MODIA  - the  alternative  course  designs  - are  very  satisfac- 
tory, the  planning  system  itself  has  some  limitations  which  degrade  its  acceptability  to 
course  planners.  Some  of  these  limitations  are  minor,  such  as  not  having  the  option  to  cancel 
certain  printouts  or  being  forced  to  go  off-line  and  then  return  to  make  certain  corrections 
to  inputs.  Others  are  more  serious,  such  as  the  garbling  of  resources  discussed  earlier. 

Rand  personnel  researched thisproblem  using  their  own  IBM  370/158  computer  and  concluded  that 
the  garbling  was  external  to  the  MODIA  programs.  Regardless  of  the  source,  the  problem  could 
have  serious  consequences.  During  the  service  test,  the  planners  were  motivated  to  repeat 
those  phases  of  course  design  where  the  garbling  occurred  until  the  problem  cleared  up.  During 
normal  operation,  this  persistence  might  not  be  in  evidence.  This  frustrating  problem  needs 
to  be  solved. 


Another  more  serious  problem  concerns  program  limitations.  In  its  current  version , MODIA 
permits  planners  to  designate  only  31  resources  (21  "special"  and  10  "other").  As  already 
pointed  out,  for  very  large  courses,  such  as  3AQR30020-1  where  many  trainers  are  located  in 
dozens  of  classrooms,  this  limitation  prohibits  planners  from  simulating  the  entire  course. 

The  course  can  be  simulated  by  blocks  of  material,  however,  but  this  is  a very  lengthy  process. 
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This  resource  limitation  is  also  a problem  for  the  standard  "sets"  or  equipment 
courses.  For  instance,  planners  for  Course  3ABR30434  identified  approximately  125 
different  pieces  of  hardware  used  to  teach  this  14-week  course.  Since  many  of  these 
pieces  of  equipment  are  always  used  together,  they  were  packaged  together  as  one  resource. 
Through  this  procedure,  the  planners  were  able  to  exercise  M0D1A  to  simulate  course 
operation  using  18  resourc?  groups.  As  might  be  expected  with  this  procedure,  the 
percentage  of  resource  utilization  given  by  MODIA  is  somewhat  in  error.  Pieces  of  equip- 
ment within  each  of  the  18  resource  groups  are  not  used  at  uniform  rates.  However, 
planners  believe  the  utilization  rates  provide  sufficient  accuracy  for  planning  purposes. 

Many  planners  felt  that  this  program  limitation  was  a real  hindrance  to  planning. 

They  either  could  not  package  resources  logically  or  failed  to  identify  the  strategy  which 
would  permit  them  to  do  so.  Moreover,  the  actual  number  of  special  resources  available 
to  a planner  is  further  reduced  by  the  manner  in  which  a given  resource  is  used  to  support 
more  than  one  learning  event. 

For  example,  to  generate  a MODIA  alternative  for  Course  3ABR30434,  planners  had  to 
designate  one  resource  by  two  labels;  the  AN/GRC-27  Radio  Set  was  designed  as  both  a 
transmitter/modulator  (Lab  3A)  and  a receiver  (Lab  3B).  Block  XIII  of  this  self-paced 
course  was  broken  down  into  14  separate  units  of  instruction.  These  14  units  alternated 
between  theory  (self-study)  and  practice  (lab  exercises),  the  latter  performed  on  either 
the  transmitter/modulator  or  the  receiver  where  both  can  be  operated  independently.  Since 
both  Lab  3A  and  Lab  3B  can  handle  two  students  at  a time,  each  complete  Radio  Set  Trainer 
can  handle  four  students  at  one  time.  For  this  alternative,  Lab  3A  (the  xmtr/md)  supported 
Learning  Events  35,  37,  and  39;  Lab  3B  (rcvr)  supported  Learning  Events  41,43,45,  and  47. 

By  breaking  the  Radio  Set  Trainer  into  two  resources,  the  planners  enabled  MODIA  to 
corrently  determine  the  number  of  trainers  required  to  eliminate  student  queuing.  If  the 
trainer  had  been  entered  into  MODIA  as  one  resource  with  a capacity  of  four  students, 

MODIA  would  have  erroneously  programmed  four  students  per  trainer  for  all  the  learning 
events,  although  the  trainer  capacity  for  the  identified  learning  events  is  actually  two. 
If  the  resource  had  been  treated  as  a single  unit,  which  it  is,  MODIA  would  have  projected 
lower  equipment  requirements  than  are  actually  required  to  handle  a given  student  flow. 

b.  The  final  limitation  deals  with  the  availability  of  computer  hardware  to  support 
MODIA.  Rand  Working  Notes  have  stressed  that  if  MODIA  is  to  be  a viable  system  foe 
course  planners,  it  should  be  both  responsive  and  accessible.  The  latest  Work  Note^  even 
establishes  time  standards  for  various  phases  of  MODIA  operation,  contending  that  if  these 
standards  are  not  met  "the  interactive,  iterative  nature  of  the  design  process  will  be 
destroyed."  The  same  Working  Note  argues  for  the  use  of  one  or  more  remote  terminals 
located  in  a working  area  away  from  the  computer  center  and  near  materials  relevant  to 
course  planning. 

During  the  service  test,  MODIA  was  exercised  through  a remote  terminal  and  on  many 
occasions  the  response  standards  discussed  by  Rand  were  achieved.  However,  because  of 
MODIA's  considerable  core  requirements,  very  tight  scheduling  was  necessary  between  the 
MODIA  planners  and  the  other  users  of  the  H-6060  computer.  When  the  Interface  Team 
realized  a certain  phase  of  MODIA  simulation  would  not  be  completed  during  the  allotted 
time,  a telephone  call  was  made  to  the  computer  center  to  request  a time  extension.  In 
this  manner,  other  programs  did  not  "dump"  MODIA  off-line  prior  to  closing  a file. 

Because  MODIA  was  being  exercised  during  a service  test  - a service  test  which  had 
considerable  priority  - other  users  of  the  Honeywell  computer  made  acconmodations  to  the 
MODIA  planners.  Without  these  accommodations,  MODIA  would  have  been  neither  accessible 
nor  responsive.  Finally,  to  the  extent  that  comparable  and  available  computer  hardware  is 
not  available  at  the  other  centers,  MODIA  cannot  be' operated  under  the  same  concept  as  it 
was  operated  at  Keesler. 

T.  MODIA:  A Tool  for  Planning  the  Use  of  Air  Force  Training  Resources  - Vol  1,  Overview, 
June  1976. 
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c.  Although  the  MODIA  approach  to  course  planning  looks  promising,  it  must  be 
remembered  that  these  conclusions  are  based  upon  rather  limited  planning  exercises. 

ISD  teams  were  not  guided  by  the  policies  now  receiving  priority  from  training 
managers:  1)  Achievement  of  a 10%  reduction  in  Instructor  manning,  2)  Reduction  of  staff 
overhead,  3)  Energy  conservation,  4)  Elimination  of  low  flow  courses,  and  5)  Reduction  of 
course  length  via  the  Hasty  Spark  Program.  Rather,  ISD  teams  were  challenged  to  experiment 
in  course  design  to  maximize  course  operation  in  whatever  dimension  they  chose.  Further, 
with  the  exception  of  the  Lowry  planners,  these  teams  knew  that  the  courses  they  designed 
would  not  be  implemented  and  no  actual  procurement  or  transfer  of  resources  would  be  re- 
quired. Finally,  again  it  should  be  pointed  out  that  these  conclusions  are  based  upon 
planning  only  six  courses,  one  of  which  could  not  be  completely  simulated. 

In  addition,  the  service  test  pointed  out  that  MODIA's  value  as  a planning  tool  could 
be  reduced  without  overall  direction  and  control  by  management.  With  a "course  perspective" 
planners  might  be  tempted  to  implement  designs  which  optimize  dimensions  of  course  opera- 
tion important  to  them  but  degrade  overall  management  efficiency.  For  example,  planners 
for  Course  3ABR30333  found  that  reduced  hardware  improved  resource  utilization,  reduced 
operating  cost,  but  extended  course  length.  If  this  design  were  implemented,  instructors 
would  be  dedicated  to  the  course  for  a longer  period  of  time  and  thus  less  available  to 
support  the  new  "family  grouping"  concept  for  instructor  manning.  Thus,  maximization  of 
MODIA  will  depend  upon  sound  management  of  the  system;  however,  the  service  test  did  not 
attempt  to  establish  how  this  system  should  be  managed. 

SECTION  E - CONCLUSIONS 

18.  Based  on  the  findings  from  this  service  test,  the  following  conclusions  can  be  drawn: 

a.  MODIA  System  Cost  for  Course  Planning:  Counting  computer  operation  costs,  the 
MODIA  approach  to  course  planning  is  more  expensive  than  the  conventional  manual  approach. 
However,  from  the  limited  data  collected,  it  appears  that  planning  with  MODIA  can  result 
in  more  cost-effective  instructional  systems  than  conventional  planning.  Further,  these 
potential  savings,  when  projected  to  the  population  of  courses  to  which  MODIA  can  be 
applied,  appear  to  be  greater  than  the  operational  cost  of  MODIA. 

b.  MODIA  System  Validity:  MODTA  programs  address  all  the  planning  factors  which 
must  be  addressed  during  course  revision/implementation.  The  programs  can  be  utilized  on  a 
computer  system  with  characteristics  similar  to  the  Honeywell  H-6060.  When  compared  to 
parameters  of  actual  ongoing  courses,  the  Resource  Utilization  Model  (RUM)  and  the  Cost 
Model  outputs  possess  a great  deal  of  fidelity.  Both  ISD  team  members  and  Interface  Team 
members  were  confident  that  simulated  course  operations  and  functional/program  cost 
printouts  were  accurate. 

c.  MODIA  System  Effectiveness  as  a Planning  Tool: 

(1)  Results  indicate  that  planning  with  MODIA  will  enable  course  planners  to 
accomplish  the  following: 

(a)  Rapidly  design  many  instructional  system  alternatives. 

(b)  Identify  critical  elements  of  a given  course  design  and  revise  the  design 
to  assess  the  impact  on  these  elements. 

(c)  Optimize  a given  course  design  with  respect  to  cost,  number  of  students 
trained,  instructor  overhead,  resource  utilization,  student  queue,  proficiency  level  of 
training,  and  so  fprth. 

(d)  Examine  the  impact  of  innovations  in  course  design. 
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For  instance,  different  approaches  to  instruction  such  as  group-pacing  or  self-pacing  or 
variations  in  content  treatment  can  be  simulated.  Conservative  planning  and  reliance  on 
previous  methodologies  need  not  take  precedence. 

(2)  For  very  large  courses  with  several  planning  factors,  such  as  keesler's 
3AQR30020-1 , Electronic  Principles  Course,  MODIA  RUM  limitations  prohibit  simulation  of 
course  operation.  However,  such  courses  can  be  simulated  by  blocks  of  instruction. 

d.  MODIA  System  Usefulness:  Technical  training  center  personnel  could  not  use  MODIA 
to  plan  Types  1,  5 and  6 courses  since  these  courses  are  planned  and  conducted  by  agencies 
other  than  the  centers.  Further,  because  of  their  duration,  supporting  equipment,  and 
purpose.  Types  2 and  4 courses  would  be  only  partially  amenable  to  the  MODIA  approach. 

However,  MODIA  is  well  suited  to  planning  and  revising  Type  3 courses,  and  since  the 
majority  of  planning  at  a center  deals  with  Type  3 courses  (SO  at  keesler)  MODIA  would 
have  considerable  utility. 

e.  MODIA  System  Acceptance:  Although  some  ISD  team  planners  did  not  approve  of 
MODIA  and  several  felt  system  changes  were  required  to  make  MODIA  more  efficient,  most 
participants  in  the  service  test  reacted  very  favorably  to  MODIA.  However,  it  is  apparent 
that  enthusiastic  support  for  MODIA  will  evolve  only  if  a competent  and  innovative  Interface 
Team  exists  to  explain  the  system,  provide  guidance  and  insights,  and  interpret  the  computer 
products . 

f.  Proposed  Changes  to  Improve  MODIA  Effectiveness:  Based  on  interviews  and 
observation,  the  Interface  Team  identified  several  changes  needed  to  improve  MODIA's 
effectiveness.  In  the  area  of  program  design,  the  team  felt  that  MODIA  terminology  needs 
to  be  better  aligned  with  ISD  terminology  and  several  options  need  to  be  placed  in  the 
User  Interface  (UI)  program  to  facilitate  input  changes  and  eliminate  unnecessary  printouts. 
Further,  the  team  and  many  of  the  service  test  participants  concluded  that  the  UI  limitation 
of  31  resources  constrained  course  design.  In  the  area  of  course  operation  simulation, 
service  test  participants  felt  that  an  additional  option  needs  to  be  placed  in  the  RUM. 

This  option  should  allow  planners  to  determine  the  percentage  of  resource  utilization  for 

a given  revision  when  the  revision  is  implemented  with  students  already  in  the  existing 
course.  In  the  area  of  program  operation,  the  garbling  of  certain  letters  and  numbers 
degraded  system  usage. 

g.  MODIA  Adoption:  The  data  which  resulted  from  this  limited  tryout  of  MODIA  are 
insufficient  to  recommend  adoption  at  this  time.  It  has  not  yet  been  determined  how 
SAAS  policies  and  objectives  will  be  integrated  into  MODIA  planning,  at  what  level  the 
products  of  MODIA  will  be  used  in  decision  making,  and  how  course-by-course  MODIA  planning 
can  be  meshed  to  achieve  resource  allocation  across  a homogeneous  group  of  courses.  Further, 
neither  operating  procedures  nor  the  necessary  operational  conf iguration  to  support  MODIA 
planning  have  been  determined.  "When  these  questions  have  been  resolved,  it  appears  that 
SAAS  management  will  be  able  to  show  that  MODIA  can  improve  resource  management  in  a 
technical  training  environment." 

SECTION  F - RECOMMENDATIONS 
19.  The  following  actions  are  recommended: 

a.  Recommend  the  Rand  Corporation,  with  assistance  from  keesler,  modify  the  MODIA 
programs  to  resolve  the  problems  discussed  in  this  report. 

b.  Recommend  the  Keesler  USAFSAAS  be  tasked  to  conduct  a Phase  II  follow-on  evaluation 
of  the  MODIA  System  using  the  24  electronics  courses  associated  with  Hasty  Spark.  This 
evaluation  will  establish  both  the  procedures  and  organizational  configuration  necessary  to 
support  MODIA  planning,  and  using  realistic  management  problems,  establish  the  utility  of 
the  system  to  SAAS  management. 
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STRUCTURED  INTERVIEW  FOR  MODIA  INTERFACE  TEAM 
USAF  SCHOOL  OF  APPLIED  AEROSPACE  SCIENCES 

SERVICE  TEST  OF  THE  MODIA 
SYSTEM 


A.  General  Instructions  for  Interviewers 

Interview  MODIA  Interface  Team  as  a group.  Be  friendly  but  objective.  Do  not  judge  or 
give  advice.  Stick  to  the  format.  Probe  as  necessary  to  obtain  a full  answer.  Record  all 
Information  on  Interview  sheet.  Key  your  notes  to  the  question  asked.  Be  sure  the 
Interviewees  understand  the  purpose  of  the  Interview.  Read  and  explain  to  them  the  instruc- 
tions In  Part  B. 

B.  Instructions  to  be  Read  to  Interviewees : 


You  have  participated  as  members  of  the  MODIA  Interface  .team  responsible  for  guiding 
and  assisting  selected  1SD  teams  In  providing  course  planning  data  for  you  to  exercise 
MODIA  as  a tool  for  course  planning.  Accordingly,  you  are  in  the  best  position  to  make  a 
number  of  assessments  concerning  MODIA  operation.  This  Interview  Is  for  the  specific  pur- 
pose of  eliciting  your  responses  to  some  specific  questions  aimed  at  determining  whether 
the  MODIA  System  should  be  adopted  for  use  at  Air  Training  Command  Technical  Training 
Centers.  We’d  appreciate  your  cooperation.  Your  voluntary  conscientious  response,  with 
those  of  other  personnel,  will  be  processed  for  statistical  summaries  without  reference 
to  individuals.  Failure  to  provide  the  requested  Information  will  not  be  used  as  cause 
for  adverse  actions.  Are  you  willing  to  participate  in  this  interview?  (If  interviewees 
are  willing  to  participate,  proceed  to  the  series  of  questions.  If  they  have  any  objections, 
terminate  the  Interview). 

C.  Structured  Interview  Sheet: 

USER  INTERFACE 


1.  Did  the  program  properly  demonstrate  the  impact  of  different  sequences  on  student  flow 
and  resource  use? 

? (a)  Yes 

(b)  No 

(c)  Uncertain 

?.  Did  the  U1  give  appropriate  estimates  of  the  effects  of  teaching  method  diversification 
on  student  flow  through  the  course  and  on  requirements  for  training  resources? 

2_  (a)  Yes 

(b)  No 

(c)  Uncertain 
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3.  Did  the  UI  give  appropriate  estimates  of  the  effects  of  content  diversification  on 
student  flow  through  the  course  and  on  requirements  for  training  resources? 

_2 (a)  Yes 

(b)  No 

_ (c)  Uncertain 

4.  Was  a total  resource  assignment  report  displayed  when  you  assigned  and  identified  all 
resources  to  learning  events? 

2 (a)  Yes 

(b)  No 

5.  One  of  the  claims  of  the  designers  of  MODIA  is  that  the  only  way  to  validate  MODIA's 
usability  Is  to  see  If  the  Intended  user  can  operate  it.  Were  you  able  to  operate  the 
MOO I A System? 

? (a)  Yes 

(b)  No 

6.  Was  the  User  Interface  Guide  helpful  to  you? 

_ (a)  Very  much  so 
__  (b)  Somewhat 


2 (c)  Very  little 

(d)  Not  at  all 


7.  Could  parts  of  the  User  Interface  Guide  be  left  out? 

2 (a)  Yes 
(b)  No 

8.  If  your  answer  to  question  7 was  yes,  what  portion(s)  do  you  feel  is  (are)  irrelevant? 


9.  Was  the  User  Interface  Guide  terminology  clear  and  concise? 

(a)  Yes 

2 (b)  No 

10.  Were  difficulties  encountered  In  assigning  special  resources  to  objectives,  test  or 
reviews? 


(a)  Very  much  so 

2 (b)  Somewhat 


(c)  Very  little 

(d)  Not  at  al 1 


11.  Did  the  amount  of  time  spent  in  planning  the  course  using  MODIA  deter  you  from  trying 
alternative  methods  of  course  design? 


(a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 
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12.  Do  you  think  the  work  sheets  were  useful  In  planning  the  course? 


(a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

_2  (d)  Not  at  all 


13.  Would  a revision  of  the  work  sheets  make  the  U1  easier  to  use? 


2 (a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


14.  Did  the  program  allow  for  the  breakdown  of  objectives  In  as  small  Increments  as  you 
desire? 


(a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


15.  Do  you  think  the  User  Interface  was  useful  to  you  In  designing  a training  course? 


_ (a)  Very  much  so 
2 (b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


16.  Do  you  feel  that  you  became  better  at  using  the  User  Interface  as  you  practiced 
more? 


^ (a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


17.  Additional  Conments: 


RESOURCE  UTILIZATION  MODEL  (RUM) 


18.  Were  sections  formed  when  the  specified  number  of  students  required  were  assembled? 

2_  (a)  Yes 

(b)  No 

(c)  Uncertain 


19.  Was  student  flow  sequential  from  the  first  to  last  learning  event  as  programed? 

2 (a)  yes 

(b)  No 

(c)  Uncertain 

20.  Did  the  RUM  simulate  Instantaneous  acquisition  of  resources  as  they  were  needed? 
2_  (a)  Yes 

(b)  No 

(c)  Uncertain 

21.  Were  all  five  Recap  Reports  printed? 

2_  (a)  Yes 

(b)  No 

22.  Is  Recap  Report  5 unnecessary  since  It  Is  a sunmary  of  the  Information  given  In 
Report  4? 

(a)  Yes 

£ (b)  No 

(c)  Uncertain 

23.  Were  all  five  outputs  useful  to  you  during  the  service  test? 

2 (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

24.  Did  you  have  difficulty  in  Interpreting  the  data  In  the  RUM  outputs? 

(a)  Very  much  so  (c)  Very  little 

2 (b)  Somewhat  (d ) Not  at  all 

25.  Could  any  of  the  Simulation  Reports  be  omitted? 

(a)  Yes 

2 (b)  No 

(c)  Undecided 
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26.  It'  your  answer  to  question  25  was  yes,  what  reports  In  your  opinion  could  be  omitted? 

27.  Could  any  of  the  data  be  omitted  from  the  reports? 

(a)  Ye s 

2.  (b)  No 

(c)  Undecided 

26.  If  your  answer  to  question  27  was  yes.  what  data  In  your  opinion  could  be  omitted? 

29.  Does  any  additional  Information  need  to  be  given  In  the  output  reports? 

(a)  Yes 

2 (b)  No 

_ (c)  Undecided 

30.  If  your  answer  to  question  29  was  yes,  what  additional  information  do  you  feel 
Should  be  given  in  the  output  reports? 


31.  Old  the  working  note  for  the  Resource  Utilization  Model  give  an  accurate  description 
of  the  unit  and  provide  sufficient  depth  to  permit  implementation  and  use  at  Keesler? 

(a)  Yes 

2 ( b ) No 

(c)  Undecided 

32.  Do  you  think  the  RUM  was  useful  in  simulating  the  conducting  of  an  Instructional 
program? 


2 (a)  Very  much  so 

(c) 

Very  Little 

(b)  Somewhat 

(d) 

Not  at  all 

Do  you  think  other  course  planners 

would  benefit 

from  using  the  RUM? 

2 (a)  Very  much  so 

(c) 

Very  little 

(b)  Somewhat 

(d) 

Not  at  all 

5 
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34.  Additional  Comments: 


COST  MODEL  (MODCOM) 

35.  Were  the  input  requirements  and  available  options  sufficient  for  the  operation  of 
the  MODCOM? 

7 (a)  Yes 

(b)  No 

(c)  Undecided 

36.  Did  the  instructions  in  the  MODCOM  Manual  help  you  to  program  correct  data  input  on 
each  format  for  proper  operation  of  MODCOM? 

2 (a)  Very  much  so  (c)  Very  little 

_ (b)  Somewhat  (d)  Not  at  all 

37.  Can  the  Cost  Model  be  operated  Independent  of  the  other  components  of  the  MODIA 
System? 


2 (a)  Yes 

(b)  No 

_ (c)  Undecided 

38.  Were  all  five  outputs  produced  by  the  Cost  Model  in  accordance  with  program 
specifications? 


2 (a)  Yes 
(b)  No 

39.  Did  you  use  the  output  from  the  RUM  as  the  input  to  the  MODCOM? 
7 (a)  Yes 
(b)  No 


40.  Did  the  output  of  the  MODCOM  give  an  estimate  of  the  total  course  costs  for  a 
time  frame  up  to  five  years? 

2 (a)  Yes 

_ (b)  No 


6 
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41.  How  long  did  it  take  you  to  prepare  the  data  input  on  the  Format  Cards? 


(a)  1 to  3 hours 

y (b)  3 to  5 hours 
(c)  5. to  7 hours 

(d)  Other  (specify  the  number  of  hours) 

42.  Do  you  think  the  MODCOM  is  useful  for  determining  the  costs  of  a specific  instructional 
system? 


2 (a)  Very  much  so 

(b)  Somewhat 

43.  Additional  Comments: 


(c)  Very  little 

(d)  Not  at  all 


OVERALL  EVALUATION  OF  MODI A 

44.  Does  MODIA  give  an  accurate  estimate  of  the  resources  required  for  a course? 

? (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

45.  Does  the  MODIA  process  encourage  trying  alternative  course  designs? 

o (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

46.  Did  MODIA  allow  you  and  the  ISO  team  to  rapidly  plan  the  training  course? 

2 (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

47.  Are  opportunities  available  for  MODIA  use  when  planning  a training  course  in  a 

technical  training  environment? 

2 (a)  Yes 

(b)  No 

(c)  Undecided 
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48.  If  your  answer  to  question  ^7  was  yes,  what  Instances  do  you  feel  MODIA  would  be 
most  applicable  in  planning  training  courses  and  the  specific  type  courses  (l.e..  Types 
I,  II,  III,  IV,  V.  and  VI)? 


49.  Did  MODIA  encourage  you  and  the  planners  to  consider  the  Implications  of  the  subject 
matter  for  requirements  of  training  resources  and  teaching  strategy? 


(a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


50. 


Do  you  think  MODIA  helps  a planner  to  better  manage  his  training  resources? 

2 («)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

51.  Do  you  think  using  MODIA  unduly  complicates  the  planning  process? 

(a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  2 (d ) Not  at  all 

52.  Are  the  data  required  to  operate  MODIA  easy/difficult  to  obtain? 

(a)  Very  easy  (c)  Difficult 

(b)  Easy  (d)  Very  difficult 

53.  Do  you  think  special  skills/knowledges  are  required  to  operate  MODIA? 

(a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

54.  If  you  think  special  skills/knowledges  are  required,  what  are  they? 
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55.  In  comparison  to  the  conventional  mode  of  planning,  did  you  feel  better  able  to  plan 
training  courses  when  you  were  using  MODIA. 


(a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


56.  Do  you  feel  planning  a training  course  can  be  done  equally  as  well  using  conventional 
means  as  using  MODIA? 


(a)  Very  much  so 

(b)  Somewhat 


2 (c)  Very  little 

(d)  Not  at  all 


57.  Did  you  feel  confident  in  the  results  obtained  by  MODIA  planning? 

_2 (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

58.  Given  a choice  of  selecting  MODIA  or  the  conventional  method  for  planning  a training 
course,  which  would  you  use  to  plan  the  course? 


2 (a)  MODIA 

_ (b)  Conventional  mode 

59.  Do  you  think  planning  a training  course  will  be  more  enjoyable  when  you  are  using 
MODIA? 


_2 (a)  Very  much  so  (c)  Very  little 

(b)  Somewhat  (d)  Not  at  all 

60.  Should  you  feel  that  MODIA  can  be  used  effectively  for  planning  training  courses, 
which  organization  should  Implement  and  operate  the  system? 

_ (a)  Training  Plans  Branch  at  the  Tech  School 

_ (b)  Training  Groups  of  the  School 

2 (c)  A specially  trained  team  operating  out  of  Tech  School 

61.  If  your  answer  to  question6Q  was  "c,"  what  should  be  the  makeup  of  the  MODIA 
team,  what  areas  of  expertise  are  required  and  at  what  levels  within  the  Technical 
School  may  they  be  found? 


9 
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62.  In  comparison  to  the  conventional  mode  of  planning,  was  course  design  time  reduced 
using  MODIA? 


(a)  Very  much  so 


2 (b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


63.  In  comparison  to  conventional  planning,  did  planning  by  MODIA  enable  you  to  consider 
more  alternatives? 

_2 (a)  Yes 

(b)  No 

64.  List  any  changes  to  MODIA  you  feel  might  Improve  Its  effectiveness. 


65.  In  comparison  to  using  MODIA  In  planning  the  revision  of  an  existing  course,  do  you 
feel  that  It  can  be  used  as  effectively  In  planning  a new  course? 


2 (a)  Very  much  so 

(b)  Somewhat 


(c)  Very  little 

(d)  Not  at  all 


66.  Other  Comments:  Please  provide  any  additional  comments  you  feel  will  aid  the 
evaluators  in  assessing  the  value  of  MODIA  in  a technical  training  setting. 


DEPARTMENT  OF  THE  AIR  FORCE 
KEESLER  TECHNICAL  TRAINING  CENTER 
KEESLER  AIR  FORCE  BASE,  MISSISSIPPI 

STRUCTURED  INTERVIEW  FOR  ISD  TEAMS 
USAF  SCHOOL  OF  APPLIED  AEROSPACE  SCIENCES 
SERVICE  TEST  OF  THE  MODIA  SYSTEM 

A.  GENERAL  INSTRUCTIONS  FOR  INTERVIEWERS:  Interview  ISD  team  members  as  a group  in  their 
respective  team.  Be  friendly  but  objective.  Do  not  judge  or  give  advise.  Stick  to  the 
format.  Probe  as  necessary  to  obtain  a full  answer.  Record  all  Information  on  Interview 
sheet.  Key  your  notes  to  the  question  asked.  Be  sure  the  Interviewees  understand  the 
purpose  of  the  Interview.  Read  and  explain  to  them  the  Instructions  In  Part  B. 

B.  INSTRUCTIONS  TO  BE  READ  TO  INTERVIEWEES:  You  have  participated  as  members  of  an  ISD 
team  in  providing  course  planning  data  for  the  MODIA  Interface  team  to  exercise  the 
MODIA  System  In  planning  your  training  course.  Accordingly,  you  are  In  the  best  position 
to  make  a number  of  assessments  concerning  MODIA  requirements  for  operation.  This  Inter- 
view is  for  the  specific  purpose  of  eliciting  your  responses  to  specific  questions  aimed 
at  determining  whether  the  MODIA  System  should  be  adopted  for  use  at  Air  Training  Command 
Technical  Training  Centers.  We'd  appreciate  your  cooperation.  Your  voluntary  conscien- 
tious response,  with  those  of  other  personnel  will  be  processed  for  statistical  summaries 
without  reference  to  individuals.  Failure  to  provide  the  requested  Information  will  not 
be  used  as  cause  for  adverse  actions.  Are  you  willing  to  participate  in  this  interview? 
(If  interviewees  are  willing  to  participate,  proceed  to  the  series  of  questions.  If  they 
have  any  objections,  terminate  the  interview.) 


C.  STRUCTURED  INTERVIEW  SHEET: 


USER  INTERFACE  (UI) 


1.  Did  the  program  allow  for  as  many  teaching  methods  as  you  would  like  to  use? 


12 

(a) 

Very  much  so 

5 

(b) 

Somewhat 

1 

(c) 

Very  little 

1 

(d) 

Not  at  all 

2.  Were  you  able  to  assign  all  training  content  of  your  training  course  to  the  subject 
matter  types  given  in  the  User  Interface  program? 


(a) 

Very  much  so 

4 

(b) 

Somewhat 

1 

(c) 

Very  little 

1 

(d) 

Not  at  all 

3.  Did  the  program  allow  you  to  designate  the  required  resources  for  objectives,  tests 
and  reviews  with  subject  matter  types? 


_5_ 

(a) 

Very  much  so 

12 

(b) 

Somewhat 

2 

(c) 

Very  little 

(d) 

Not  at  all 

4.  Did  the  UI  provide  sufficient  options  for  you  to  describe  student  arrivals  In  your 
training  course  In  order  to  measure  significant  effects  on  the  use  of  training  resources? 


.10- 

(a) 

Very  much  so 

7 

(b) 

Somewhat 

2 

(c) 

Very  little 

(d) 

Not  at  all 

5.  Was  the  UI  useful  in  giving  appropriate  estimates  of  the  effects  of  teaching  method 
diversification  on  student  flow  through  your  course  and  on  requirements  for  training 
resources? 

6 (a)  Very  much  so 

8 (b)  Somewhat 

1 (c)  Very  little 

(d)  Not  at  all 


if  tJ/ft 


6.  Was  the  UI  useful  in  giving  appropriate  estimates  of  the  effects  of  content 
diversification  on  student  flow  through  the  course  and  on  requirements  for  training 
resources? 

5 (a)  Very  much  so 

ft  (b)  Somewhat 

(c)  Very  little 

(d)  Not  at  all 


6 N/A 


7.  Did  the  amount  of  time  spent  in  planning  your  course  using  the  UI  deter  you  and  the 
interface  team  from  trying  alternative  methods  of  course  design? 


_6_ 


(a) 

Very  much  so 

(b) 

Somewhat 

(0 

Very  little 

(d) 

Not  at  all 
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8.  Did  the  UI  work  sheets  have  to  be. revised  for  you  to  provide  the  appropriate  data  to 


exercise 

MOD I A? 

3 

(a) 

Very  much  so 

-6_ 

(b) 

Somewhat 

_a_ 

(c) 

Very  little 

(d) 

Not  at  all 

9.  Can  you  suggest  additional  work  sheets  that  would  make  the  UI  easier  to  use? 


10.  Did  you  have  difficulty  In  providing  the  appropriate  planning  factors  required  to 


exercise 

the  UI 

program? 

4 

(a) 

Very  much  so 

5 

(b) 

Somewhat 

1 

(c) 

Very  little 

_9 

(d) 

Not  at  all 

11.  Did  the  UI 

program  allow 

as  you  desire? 

_a__ 

(a) 

Very  much  so 

11 

(b) 

Somewhat 



(c) 

Very  little 

(d) 

Not  at  all 

12.  Do  you  think  the  User  Interface  was  useful  to  you  In  designing  your  training  course? 

_8 (a)  Very  much  so 

_5 (b)  Somewhat 

(c)  Very  little 

(d)  Not  at  all 
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13.  Old  you  learn  anything  useful  from  using  the  User  Interface? 


10 

(a) 

Very  much  so 

s 

(b) 

Somewhat 

4 

(c) 

Very  little 

(d) 

Not  at  all 

14.  Would  you  like  to  make  any  additional  comments  concerning  the  User  Interface 
component? 


RESOURCE  UTILIZATION  HOPEI  (RUM) 

15.  Was  simulation  of  your  training  course  terminated  after  a specified  number  of  course 
hours  and  specified  number  of  student  graduates? 

lfi  (a)  Yes 

3 (b)  No 

16.  Is  Recap  Report  5 necessary  since  it  is  a sumnary  of  the  information  given  in  Report 
4? 

11  (a)  Yes 

7 (b)  No 

(c)  Uncertain 

17.  Are  all  five  outputs  necessary? 

14  (a)  Yes 

5 (b)  No 

(c)  Uncertain 

18.  Were  the  data  in  the  RUM  outputs  easy  to  interpret? 

7 (a)  Yes 

12  (b)  No 
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19.  Do  you  think  the  RUM  was  useful  In  simulating  your  training  course? 

7 (a)  Very  much  so 

8 (b)  Somewhat 

•_  (c)  Very  little 

4 (d)  Not  at  all 

20.  Additional  Corments: 


COST  MODEL  (MODCOM) 

21.  Do  the  instructions  in  the  MODCOM  Manual  need  to  be  revised  in  order  for  you  to  provide 
correct  data  input  for  each  format? 

3 (a)  Very  much  so 

3 (b)  Somewhat 

^ (c)  Very  little 

___  (d)  Not  at  all  8 N/A 

22.  How  long  did  it  take  to  collect  the  data  required  for  inputs  to  the  format  Cards? 

(a)  Less  than  1 hour 

4 (b)  1 to  3 hours 

7 (c)  3 to  5 hours 

(d)  5 to  7 hours 

^2  (e ) Other  (specify  the  number  of  hours)  (9  to  40  Hours) 

23.  Do  you  think  the  MODCOM  is  useful  for  determining  the  costs  of  an  instructional  system? 


-10-- 

(a) 

Very  much  so 

_6 

(b) 

Somewhat 

3 

(0 

Very  little 

(d) 

Not  at  all 
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24.  Make  any  other  comments  you  would  like  to  about  this  component  and  give  to  the  inter- 
viewer. 


OVERALL  EVALUATION  OF  MODI A 

25.  In  your  opinion,  does  MODIA  give  an  accurate  estimate  of  the  resources  required  for 
a course? 

8 (a)  Very  much  so 

7 (b)  Somewhat 

2 (c)  Very  little 

2 (d)  Not  at  all 

26.  Does  the  MODIA  process  encourage  trying  alternative  course  designs? 

8 (a)  Very  much  so 

6 (b)  Somewhat 

2 (c)  Very  little 

q (d)  Not  at  all 

27.  Does  MODIA  allow  the  user  to  rapidly  plan  his  training  course? 

10  (a)  Yes 

9 (b)  No 

28.  Are  opportunities  available  for  MODIA  use  when  planning  a training  course  in  a 
technical  training  environment? 


13 

(a) 

Yes 

4 

(b) 

No 

_2_ 

(c) 

Uncertain 

6 
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29.  If  your  answer  to  question  28  was  yes,  list  as  many  Instances  that  you  feel  MODIA 
would  be  most  applicable  In  planning  training  courses  and  the  specific  type  courses 
(!.e..  Types  I,  II,  III,  IV,  V,  and  VI). 


30.  Does  MODIA  encourage  a planner  to  consider  the  Implications  of  the  subject  matter 
for  requirements  for  training  resources  and  teaching  strategy? 

(a)  Very  much  so 

(b)  Somewhat 

_3 (c)  Very  little 

10  (d)  Not  at  all 

31.  Does  MOO I A allow  a planner  to  better  plan  his  training  resources? 

10 (a)  Very  much  so 

(b)  Somewhat 

(c)  Very  little 

_4 (d)  Not  at  all 

32 


Does  MODIA  unduly  complicate  the  planning  process? 

Jj (a)  Very  much  so 

_2 (b)  Somewhat 

_6 (c)  Very  little 

_1 (d)  Not  at  all 

33.  Did  you  have  difficulty  In  gathering  the  data  required  to  operate  MODIA? 


(a) 

Very  much  so 

6 

(b) 

Somewhat 

2 

(c) 

Very  little 

7 

(d) 

Not  at  all 
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34.  Do  you  think  special  skills/knowledges  are  required  to  operate  MODIA? 
7 (a)  Very  much  so 

22  (b)  Somewhat 

1 (c)  Very  little 

(d)  Not  at  all 


35.  If  you  answered  a or  b to  question  34  what.  In  your  opinion,  are  the  skills/knowl- 
edges? 


36.  In  comparison  to  the  conventional  mode  of  planning,  do  you  feel  better  able  to  plan 
training  courses  using  MODIA? 


5 

(a) 

Very  much  so 

_5_ 

(b) 

Somewhat 

5 

(c) 

Very  little 

-A- 

(d) 

Not  at  all 

37.  Do  you  feel  planning  a i 
means  as  using  MODIA? 

(a) 

Very  much  so 

_9_ 

(b) 

Somewhat 

2 

(c) 

Very  little 

2 

(d) 

Not  at  all 

38.  Do  you  feel  confident  ii 

JL_ 

(a) 

Very  much  so 

10 

(b) 

Somewhat 

3 

(c) 

Very  little 

(d) 

Not  at  all 
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39.  Should  you  feel  that  MODIA  can  be  used  effectively  for  planning  training  courses, 
which  organization  should  Implement  and  operate  the  system? 

(a)  Training  Plans  Branch  at  the  Tech  School 

(b)  Training  Groups  of  the  School 

(c)  A specially  trained  teem  operating  out  of  Tech  School 

40.  If  your  answer  to  question  39  was  "c,"  what  should  be  the  makeup  of  the  MCDIA 
team,  what  areas  of  expertise  are  required  and  at  what  levels  within  the  Technical 
School  may  they  be  found? 


41.  List  any  changes  In  your  opinion  which  might  Improve  MODIA's  effectiveness: 


42.  In  comparison  to  using  MODIA  in  planning  the  revision  of  an  existing  course,  do  you 
feel  that  It  can  be  used  as  effectively  in  planning  a new  course? 

\2 (a)  Very  much  so 

\ 

(b)  Somewhat 

JJ (c)  Very  little 

_ (d)  Not  at  all 

43.  Other  Comnents:  Please  provide  any  additional  comments  you  feel  will  aid  the 
evaluators  in  assessing  the  value  of  MODIA  in  a technical  training  setting. 
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HISTORY  SHEET 

A.  INTRODUCTION:  You  have  participated  In  the  evaluation  of  the  MODIA  System  at 
Keesler  during  the  period  8 March  - 4 Juno  1976;  therefore,  you  are  in  the  best 
position  to  make  a number  of  assessments  concerning  the  usefulness  of  MODIA  as  a 
tool  for  planning  a training  course.  This  historical  sheet  has  been  designed  to 
assist  you  in  furnishing  us  with  specific  Information  relative  to  your  background 
Please  answer  each  question  and  return  the  history  sheet  to  the  project  monitor. 

B.  BACKGROUND:  4 


1.  Sex:  Male_ 

2.  Grade/Rank: 

3.  Education: 


Female 


Years  High  School 


Years  College 


Degree  (Indicate  Type) 
Group  Membership:  


What  is  your  job/position  title? 


6.  How  long  have  you  worked  in  your  present  or  similar  position? 

7.  Experience: 

a.  Training  Course  Planning: Years 

b.  Instructor:  Years 

c.  Other:  

Federal  Service  At  Keesler  AFB 


.Years 


8.  Total  Years: 


ESTIMATED  COST  FOR  MANUALLY  PLANNING  FIVE  KEESLER 
COURSES  SELECTED  FOR  THE  SERVICE  TEST 

COURSE  3ABR32834 


No.  Course  Hours  = 510 


Grade 

Rate 

Cost 

GS-11 

11.0  Hrs 

$ 9.12/Hr 

$ 100.32 

GS-11 

35.5  Hrs 

$10. 16/Hr 

$ 360.68 

GS-11 

26.0  Hrs 

$ 9.38/Hr 

$ 243.86 

GS-11 

66.9  Hrs 

$ 9.64/Hr 

$ 636.24 

GS-12 

12.0  Hrs 

$12. 12/Hr 

$ 145.44 

E-7 

94.0  Hrs 

$ 6.90/Hr 

$ 648.60 

E-7 

94.0  Hrs 

$ 6.90/Hr 

$ 648.60 

E-7 

94.0  Hrs 

$ 6.90/Hr 

$ 648.60 

E-6 

250.0  Hrs 

$ 5.93/Hr 

$1,482.50 

E-6 

250.0  Hrs 

$ 5.93/Hr 

$1,482.50 

E-6 

250.0  Hrs 

$ 5.93/Hr 

$1,482.50 

Total 

994.5  Hrs 

- 

$6,582.64 

* No.  manhours  expended  are  professional  estimates  of  personnel  directly  responsible 
for  the  last  revision  of  Course  3ABR32834. 


COURSE  3ABR30331 


No.  Course  Hours  = 690 


Grade 

* Manhours 

Expended 

Rate 

Cost 

GS-11 

120  Hrs 

$ 9.64/Hr 

$1,156.80 

GS-11 

4 Hrs 

$ 9.64/Hr 

$ 38.56 

GS-11 

4 Hrs 

$ 9.38/Hr 

$ 37.52 

GS-11 

320  Hrs 

$ 8.68/Hr 

$2,835.20 

GS-9 

320  Hrs 

$ 7.99/Hr 

$2,556.80 

GS-12 

4 Hrs 

$11. 18/Hr 

$ 44.72 

E-7 

320  Hrs 

$ 6.90/Hr 

$2,208.00 

E-7 

20  Hrs 

$ 6.90/Hr 

$ 138.00 

Total 

1,112  Hrs 

- 

$9,015.60 

* No.  manhours  expended  are  professional  estimates  of  personnel  directly  responsible 


for  the  last  revision  of  Course  3ABR30331. 
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COURSE  3ABR30333 


No.  Course  Hours  = 570 


Grade 

* Manhours 
Expended 

Rate 

Cost 

GS-11 

50  Hrs 

$ 9.64/Hr 

$ 482.00 

GS-11 

8 Hrs 

$ 9.64/Hr 

$ 77.12 

GS-11 

8 Hrs 

$ 9.90/Hr 

$ 79.20 

GS-11 

25  Hrs 

$ 9.38/Hr 

$ 234.50 

GS-12 

15  Hrs 

$11. 80/Hr 

$ 177.00 

E-6 

50  Hrs 

$ 5.93/Hr 

$ 296.50 

E-6 

440  Hrs 

$ 5.93/Hr 

$2,609.20 

E-6 

440  Hrs 

$ 5.93/Hr 

$2,609.20 

Total 



1,036  Hrs 

- 

$6,564.72 

L _ . | . . - i 

* No.  of  manhours  expended  are  professional  estimates  of  personnel  directly  responsible 


for  the  last  revision  of  Course  3ABR30333. 

COURSE  3A3R30434 


No.  Course  Hours  = 420 


Grade 

lSSSSSHHH 

Rate 

Cost 

GS-12 

4 Hrs 

$10. 56/Hr 

$ 42.24 

GS-11 

120  Hrs 

$ 8.86/Hr 

$1,063.20 

GS-11 

8 Hrs 

$ 9.64/Hr 

$ 77.12 

GS-11 

8 Hrs 

$ 8.86/Hr 

$ 70.88 

GS-12 

8 Hrs 

$10. 87/Hr 

$ 86.96 

GS-12 

8 Hrs 

$11. 49/Hr 

$ 91.92 

GS-11 

16  Hrs 

$ 8.60/Hr 

$ 137.60 

GS-11 

16  Hrs 

$ 8.60/Hr 

$ 137.60 

GS-9 

140  Hrs 

$ 7.13/Hr 

$ 998.20 

GS-9 

140  Hrs 

$ 7.78/Hr 

$1,089.20 

GS-11 

60  Hrs 

$ 9.64/Hr 

$ 478.40 

E-7 

16  Hrs 

$ 6.90/Hr 

$ 110.40 

E-7 

16  Hrs 

$ 6.90/Hr 

$ 110.40 

E-6 

140  Hrs 

$ 5.93/Hr 

$ 830.20 

Total 

- 1 

700  Hrs 

- 

$5,424.32 

* No.  of  manhours  expended  are  professional  estimates  of  personnel  directly  responsible 
for  the  last  revision  of  Course  3ABR30434. 
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COURSE  3AQR30020-1 


No.  Course  Hours  = 540 


Grade 

* Manhours 

Expended 

Rate 

Cost 

GS-11 

304  Hrs 

$ 8.86/Hr 

$ 2,693.44 

GS-11 

20  Hrs 

$ 9.64/Hr 

$ 192.80 

GS-11 

20  Hrs 

$ 8.86/Hr 

$ 177.20 

GS-12 

20  Hrs 

$10. 87/Hr 

$ 217.40 

GS-12 

20  Hrs 

$11. 49/Hr 

$ 229.80 

GS-12 

10  Hrs 

$10. 56/Hr 

$ 105.60 

GS-11 

153  Hrs 

$ 9.64/Hr 

$ 1,474.92 

GS-y 

303  Hrs 

$ 7.13/Hr 

$ 2,160.39 

GS-9 

303  Hrs 

$ 7.78/Hr 

$ 2,357.34 

GS-11 

82  Hrs 

$ 8.60/Hr 

$ 705.00 

E-7 

82  Hrs 

$ 6.90/Hr 

$ 656.80 

E-6 

303  Hrs 

$ 5.93/Hr 

$ 1,796.79 

Total 

1620  Hrs 

- 

$12,676.68 

* No.  manhours  expended  are  professional  estimates  of  personnel  directly  responsible  for  the 
last  revision  of  Course  3AQR30020-1. 
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USAF  COST  INCURRED  TO  OPERATE  MOOIA  DURING  SERVICE  TEST 


COURSE  3ABR32834 


No.  Course  Hours  = 510 


Computer  Utilization 


Supplies  Used 


GS-13 


GS-12 


E- 


GS-12 


GS-12 


GS-12 


E- 


GS-9 


GS-9 


GS-11 


E- 


E-7 


1 Part  Paper 


Thermographic 

Paper 


Bell  103A  Data  Set  Rental 


Cost  for  preparing  two  alternatives. 


No.  Manhours  Expended  = 484 


Factor 

Rate 

Cost 

3,030  Min 

$1. 59/Mi  n- 

$4,817.70 

2 Hrs 

$12. 48/Hr 

$24.96 

4 Hrs 

$10. 56/Hr 

$42.24 

121  Hrs 

$6. 90/hr 

$834.90 

2 Hrs 

$10. 56/Hr 

$21.12 

2 Hrs 

$10. 16/Hr 

$20.32 

2 Hrs 

$10. 16/Hr 

$20.32 

80.0  Hrs 

$6. 90/ Hr 

$552.00 

194.5  Hrs 

$7. 56/Hr 

$1,470.42 

60.5  Hrs 

$7. 78/Hr 

$470.69 

8 Hrs 

$10. 16/Hr 

$81.28 

4 Hrs 

$6. 90/Hr 

$27.60 

4 Hrs 

$6. 90/Hr 

$27.60 

1 Case 

$20. 00/Case 

$20.00 

5 Rolls 

$5. 00/Roll 

$25.00 

0.6  Mo 

$53. 92/Mo 

$32.35 

*Total  Cost 

$8,488.50 
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No.  Course  Hours=420 


Computer  Utilization 


GS-13 


Personnel  Usage  E-9 
E-6 


Supplies  Used 


MODEM  Rental 


GS-11 


GS-11 


GS-9 


GS-9 


GS-9 


1 Part  Paper 


Thermographic 

Paper 


COURSE  3ABR30434 


Factor 

Rate 

' Cost 

3,153  Min 

$1^  59/MI  n 

$5,013.27 

10  Hrs 

$124.80 

120.0  Hrs 

$10. 56/Hr 

$1,267.20 

1.5  Hrs 

$4. 34/Hr 

$6.51 

7.0  Hrs 

$6. 90/Hr 

$48.30 

2.0  Hrs 

$10. 56/Hr 

$21.12 

2.0  Hrs 

$10. 25/Hr 

$20.50 

7.0  Hrs 

$9. 15/Hr 

$64.05 

149.0  Hrs 

$5. 93/Hr 

$883.57 

3.0  Hrs 

$8. 60/Hr 

$25.80 

68.5  Hrs 

$8. 86/Hr 

$606.91 

91.0  Hrs 

$7. 13/Hr 

$648.83 

4.5  Hrs 

$7. 35/Hr 

$33.08 

104.0  Hrs 

$7. 78/Hr 

$809.12 

1 Case 

$20. 00/Case 

$20.00 

4 Rolls 

$5. 00/Roll 

$20.00 

0.6  Mo 

$53. 92/Mo 

$32.35 

* Total  Cost 

$9,645.41 

* Cost  for  preparing  two  alternatives 
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COURSE  3ABR30331 


No.  Course  Hours  = 690 No.  Manhours  Expended  = 458,50 


Cateqory 

Factor 

Rate  1 

Cost 

Computer  Utilization 

4,055  Min 

$ 1 . 59/MI  n 

$6,447.45 

Personnel 

Usage 

GS-13 

3.0  Hr 

$12. 48/Hr 

$37.44 

GS-12 

1.0  Hr 

$10. 56/Hr 

$10.56 

E-9 

1.0  Hr 

$9. 15/Hr 

$9.15 

E-7 

124.0  Hrs 

$6. 90/Hr 

$855.60 

E-6 

111.0  Hrs 

$5. 93/Hr 

$658.23 

GS-9 

134.0  Hrs 

$7. 99/Hr 

$1,070.66 

GS-11 

56.5  Hrs 

$9. 64/Hr 

$544.66 

E-7 

20.0  Hrs 

$6. 90/Hr 

$138.00 

GS-11 

8.0  Hrs 

$8. 86/Hr 

$70.88 

Supplies  Used 

1 Part  Paper 

1.0  Case 

$20. 00/Case 

$20.00 

Thermographic  Paper 

4.0  Rolls 

$5. 00/Roll 

$20.00 

MODEM  Rental 

0.6  Mo 

$53.92  Mo 

$32.35 

* Total  Ci 

ost  $9,914.93 

* Cost  for  preparing  two  alternatives. 
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COURSE  3ABR30333 


No.  Course  Hours  - 570 No.  Manhours  Expended  = 600.5 

I 1 I Car  + nv»  1 Data  ' 1 Pnet 


Cateqory 

Factor 

Rate 

Cost 

Computer  Utilization 

2,024  Min 

$1 . 59/Min 

$3,218.16 

Personnel 

Usage 

GS-13  - 

2.0  Hrs 

$12. 48/Hr 

$24.96 

GS-12 

1.5  Hrs 

$10. 25/Hr 

$15.38 

GS-12 

144.0  Hrs 

$10. 56/Hr 

$1,520.64 

E-7 

150.0  Hrs 

$6. 90/Hr 

$1,035.00 

GS-11 

60.0  Hrs 

$9. 64/Hr 

$578.40 

E-6 

150.0  Hrs 

$5. 93/Hr 

$889.50 

GS-11 

60.0  Hrs 

$9. 38/Hr 

$562.80 

E-7 

30.0  Hrs 

$6. 90/Hr 

$207.00 

E-7 

3.0  Hrs 

$6. 90/Hr 

$20.70 

■ 

1 Part  Paper 

1.0  Case 

$20. 00/Case 

$20.00 

Thermographic  Paper 

5.0  Rolls 

$5. 00/Roll 

$25.00 

MODEM  Rental 

0.6  Mo 

$53. 02/Mo 

$32.35 

* Total  Cost  $8,072.54 


* Cost  for  preparing  two  alternatives. 
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No.  Course  Hours  = 450 


ateqor 


COURSE  3AQR30020-1 


Factor 


ost  for  preparing  two  alternatives. 


5,585  Min 


* Total  Cost 


No.  Manhours  Expended  = 693.2 


Rate  Cost 


$1.59/Min  $8,880.15 


2.00  Hrs  $12. 48/Hr 


1.25  Hrs  $9. 90/Hr 


1.5  Hrs  $10. 56/Hr 


$24.96 


$12.38 


$15.84 


$6. 90/Hr  $1,587.00 


$8. 86/Hr 


$9. 12/Hr 


$8. 86/Hr 


$203.78 


1.0  Hr  $8. 60/Hr 


1.5  Hrs  $9. 64/Hr 


0.6  Mo  $53. 92/Mo 


$32.35 


$13,792.08 


$7. 13/Hr  $1,554.34 


$6. 48/Hr  $1,380.24 


$20. 00/Case  $30.00 


6 Rolls  $5 . 00/Rol  1 $30.00 
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COURSE  3AIR75100-X  (Lowry)* 

(*  Not  Used  in  Analysis  Because  of  No  Comparative  Data) 


No.  Course  Hours  =100 


Computer  Utilization 


GS-13 


Personnel 

Usage 


Supplies  Used 


GS-12 


GS-12 


GS-11 


E- 


E- 


1 Part  Paper 


Thermographic 

Paper 


Factor 

Rate 

1,881  Min 

$1 . 59/Min 

36  Hrs 

$12. 11/Hr 

36  Hrs 

$10. 25/Hr 

36  Hrs 

$10. 56/Hr 

36  Hrs 

$5. 31/Hr 

36  Hrs 

$6. 90/Hr 

36  Hrs 

$6. 90/Hr 

0.5  Case 

$20. 00/Case 

1 Roll 

$5. 00/Roll 

■ .25  Mo 

$53. 92/Mo 

No.  Manhours  Expended  = 216 


Cost 


$435.96 


MODEM  Rental  ' .25  Mo 


Total  Cost 


NOTE:  1.  Manhours  expended  per  hour  of  course  length  = 2.16. 
2.  Cost  per  hour  of  planning  effort  = $22.65. 


$369.00 


$380.16 


$191.16 


$248.40 


$248.40 


$13.48 


$4,892.35 
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COST  DATA  FOR  MQDIA  IMPLEMENTATION 

Rate 

$1. 59/Mi n 
$12. 40/Hr 
$10. 56/Hr 
$6. 90/Hr 
$5. 93/Hr 
$5. 07/Hr 
$4. 35/Hr 
$4. 34/Hr 
$7. 99/Hr 
$7. 78/Hr 
$20. 00/Case 
$5.00/Ro1I 
J105.00/M0DEM 
$53. 92/Mo. 

$3.00 
$14.70 

Total  Cost  $36,080.25 

* Provided  by  Data  Automation  Branch  (Keesler/ACD) . 

**  Provided  by  Keesler/2052  CG.  Appendix  6 
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Telephone  *ire 

Installation  **  

Instrument 


320  Hrs 


Supplies  Used 


Computer  Paper* 


Thermographic  Paper 


7.0  Cases 


10  Rolls 


Non-Recurring  Charge  for  MODEM  Installation  1 MODEM 


MODEM  Rental 


$124.80 

$4,847.04 

$3,201.60 

$1,992.48 

$1,434.81 


$2,489.60 

$140.00 

$50.00 

$105.00 

$188.72 

$3.00 

$14.70 


Computer  Utilization* 


Factor 


11,901.27  Min 


MANHOUR  EFFORT  PER  HOUR  OF  COURSE  LENGTH 


1. 


2. 


★ 

★ * 


Manual  Planning: 

Manhours/Hr 

♦Average  Effort/ 

Course  Identity 

Course  Length  (Hrs) 

Course  Length 

•Hr  Course  Length 

3ABR32834 

510 

1.95 

3ABR30434 

420 

1.67 

3ABR30331 

690 

1.61 

3ABR30333 

570 

1.82 

3AQR30020-1 

540 

3.00 

105/5  = ■ 

or 

MODIA  Planning: 

Manhours/Hr 

**  Average  Effort/ 

Course  Identity 

Course  Length  (Hrs) 

Course  Length 

Hr  Course  Length 

3ABR32834 

510 

0.95 

3ABR30434 

420 

1.36 

3ABR30331 

690 

0.66 

3ABR30333 

570 

1.05 

3AQR30020-1 

450 

1.54 

6.56/5=  mm 

1.11 

Planned  one  instructional  alternative. 

Planned  two  instructional  alternatives  (0.55  manhours  per  hour  course  length  for 
one  alternative). 
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AVERAGE  DOLLAR  COST  TO  SUPPORT  ONE  HOUR  OF  PLANNING  EFFORT 


1.  Manually  Planning: 


Course  Identity 


Total  Manhours 
Planning  Effort 


Total 

Planning  Cost 


*Avg  Cost/ 

Cost/Hour  Hour  Planning 
Planning  Effort  Effort 


3ABR32834  994.5 
3ABR30434  700.0 
3ABR30331  1,112.0 
3ABR30333  1,036.0 
3AQR30020-1  1,620.0 


$6,582.44 

$5,424.32 

$9,015.60 

$6,564.72 

$12,676.68 


$6.62 

$7.75 

$8.11 

$6.34 

$7.83 

$36.65/5 


2.  MODIA  Planning 


Course  Identity 

3ABR32834 

3ABR30434 

3ABR30331 

3ABR30333 

3AQR30020-1 


•.ill ties.  Manhours,  and  Supplies  Cos_t) : 


Total  Manhours 
Planning  Effort 


484.00 

569.50 

485.00 

600.50 
693.27 


Total 

Planning  Cost 


$8,488.50 

$9,645.41 

$9,914.93 

$8,072.54 

$13,792.08 


Cost/Hour 
Planning  Effort 


$17.54 

$16.96 

$20.44 

$13.44 

$19.89 

$88.27/5 


**  Avg  Cost/ 
Hour  Planning 
Effort 


-$17.65 


* Planned  one  Instructional  Alternative. 

**  Planned  two  Instructional  Alternatives  ($8.83  per  hour  planning  effort  for  one 
alternative). 


FUNCTIONAL  COST  FACTORS  CONSIDERED  BY  COURSE  PLANNERS* 


Courseware  Procurement 

Printed  Media 
Display  Media 
Software  • 

Hardware  Procurement 

Media  Hardware 
Special  Equipment 
Overhead  Hardware 

Facility  Construction 

Pay  and  Allowances 

Students 
Instructors 
Curriculum  Personnel 
Hardware  Maintenance  Personnel 
Facilities  Maintenance  Personnel 
Training  Administrative  Personnel 
Base  Operating  Support  Personnel 
Medical  Personnel 

PCS  Costs 

Students 

Instructors 

TOY  Cost 

Transportation 
Destination  Per  Diem 

Instructor  Training 

Factor  Training  of  Initial  Instructor  Cadre 
Education  Training 

Miscellaneous  Operating  Costs 

Computer  Service  Charges 
Hardware  Replenishment  Repair  Parts 
Miscellaneous  Supplied 


* Adapted  from  RAND  Publication  MODIA:  The  Cost  Model,  April  1975,  RAND  Corporation, 
R-1704,  MODIA. 
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SCHEDULE  OF  TASK  FOR  MODIA  SERVICE  TEST  - 8 MARCH  THRU 
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24  May-  4 Jun  Mr  Jordan,  MSgt  Hodge  Plan  training  course  for  Lowry  AFB  3AIR75100-X 

Lowry  ISD  Team  F 


